Python Distribution

https://packaging.python.org/tutorials/managing-dependencies/

https://docs.python.org/3/distutils/index.html

Python Software Libraries
https://pypi.org/

Advertisements

Dev, QA, Prod Monitoring Tools

Problem Statement: We need to monitor services on Dev, QA, and Prod.
Customer doesn’t want to sponsor the monitoring project.

Solution:
Step 1: Monitor & Alert
Monitor service endpoints using JMeter or Python.
Write output to CSV files day wise.
Send EMails or Slack messages for given thresholds

Step 2: Aggregate Numbers
Setup Jupyter Notebook
Mount all files as input with read-only access
Write a script to tabulate and visualize data.
Protect system with id/password
https://jupyter-notebook.readthedocs.io/en/stable/public_server.html

-o-

Python Data Analysis Library – Python

https://pandas.pydata.org/
pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

Pandas visualization
http://pandas.pydata.org/pandas-docs/stable/visualization.html
-o-

HDF5
Hierarchical Data Format – Version 5
https://support.hdfgroup.org/HDF5/whatishdf5.html
We can store/retrieve simple to complex data in flat files.
Community Version Vs Enterprise Edition: https://www.hdfgroup.org/solutions/hdf5-enterprise-support-edition/

-o-

Jupyter Note Book – Interactive way to run Python code

https://jupyter-notebook.readthedocs.io/en/stable/index.html

Deploy Jupyter in Cloud: https://kyso.io/

Sample csv file
https://catalog.data.gov/dataset/annual-crime-dataset-2015

Sample Python Note Book Code
https://kyso.io/polimetla/austin-crime
-o-

Find your Mac Spec

1. Go to the Apple (top left of screen) and choose “About this Mac”.
2. Look at the pop-up window and find the serial number listed at the bottom. Copy it.
3. Go to http://www.everymac.com/ultimate-mac-lookup/ and paste your serial number where it says Enter Identifier.
4. This will bring you to a page with the model of your machine.

Reference:
https://apple.stackexchange.com/questions/268463/which-graphics-cards-does-macbook-pro-retina-15-inch-mid-2014-have

Python – Performance Tuning with Nvidia and vectorize

Step 1:
Read this first
https://weeraman.com/put-that-gpu-to-good-use-with-python-e5a437168c01

Step 2: Install NVidia in Mac
https://docs.nvidia.com/cuda/cuda-installation-guide-mac-os-x/index.html

>brew tap caskroom/drivers
>brew cask install nvidia-cuda

Then you also need to add the following to your file ~/.bash_profile:
export PATH=/Developer/NVIDIA/CUDA-9.0/bin${PATH:+:${PATH}}
export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-9.0/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}

Step 3: Try the following code

Before

import numpy as np
from timeit import default_timer as timer

def pow(a, b):
    return a ** b

def main():
    vec_size = 100000000

    a = b = np.array(np.random.sample(vec_size), dtype=np.float32)
    c = np.zeros(vec_size, dtype=np.float32)

    start = timer()
    c = pow(a, b)
    duration = timer() - start

    print(duration)

if __name__ == '__main__':
    main()
    
#performance: 2.000642750179395
    

After

import numpy as np
from timeit import default_timer as timer
from numba import vectorize

@vectorize(['float32(float32, float32)'], target='parallel')
def pow(a, b):
    return a ** b

def main():
    vec_size = 100000000

    a = b = np.array(np.random.sample(vec_size), dtype=np.float32)
    c = np.zeros(vec_size, dtype=np.float32)

    start = timer()
    c = pow(a, b)
    duration = timer() - start

    print(duration)

if __name__ == '__main__':
    main()
    
#performance: 0.4175510290078819    

—–
Reference:
https://github.com/numba/numba/issues/1898
https://stackoverflow.com/questions/38566367/installing-cuda-via-brew-and-dmg

Learn Python

https://www.tutorialspoint.com/python/index.htm

https://edu.openedg.org

Object-Oriented Programming: https://realpython.com/python3-object-oriented-programming/

Full Stack Python: https://www.fullstackpython.com/table-of-contents.html

10 Myths of Enterprise Python
https://www.paypal-engineering.com/2014/12/10/10-myths-of-enterprise-python/

Python Performance: https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/python.html

Python popularity is going up and Java popularity is going down
https://medium.freecodecamp.org/best-programming-languages-to-learn-in-2018-ultimate-guide-bfc93e615b35