Visualizing Data with Pair-Plot Using Matplotlib

2022-04-25T00:00:00+00:00

Pair Plot

A pair plot is plotting “pairwise relationships in a dataset” (seaborn.pairplot). A few well-known visualization modules for Python are widely used by data scientists and analysts: Matplotlib and Seaborn. There are many others as well but these are de facto standards. In the sense of level we can consider Matplotlib as the more primitive library and Seaborn builds upon Matplotlib and “provides a high-level interface for drawing attractive and informative statistical graphics” (Seaborn project).

Seaborn’s higher-level pre-built plot functions give us good features. Pair plot is one of them. With Matplotlib you can plot many plot types like line, scatter, bar, histograms, and so on. Pair-plot is a plotting model rather than a plot type individually. Here is a pair-plot example depicted on the Seaborn site:

Using a pair-plot we aim to visualize the correlation of each feature pair in a dataset against the class distribution. The diagonal of the pairplot is different than the other pairwise plots as you see above. That is because the diagonal plots are rendering for the same feature pairs. So we wouldn’t need to plot the correlation of the feature in the diagonal. Instead we can just plot the class distribution for that pair using one kind of plot type.

The different feature pair plots can be scatter plots or heatmaps so that the class distribution makes sense in terms of correlation. Also the plot type of the diagonal can be chosen among the mostly used kind of plots such as histogram or KDE (kernel density estimate), which essentially plots the density distribution of the classes.

Since a pair plot visually gives an idea of correlation of each feature pair, it helps us to understand and quickly analyse the correlation matrix (Pearson) of the dataset as well.

Custom Pair-Plot using Matplotlib

Since Matplotlib is relatively primitive and doesn’t provide a ready-to-use pair-plot function, we can do it ourselves in a similar way to how Seaborn does. You normally won’t necessarily create such home-made functions if they are already available in modules like Seaborn. But implementing your visualization methods in a custom way give you a chance to know what you plot and may be sometimes very different than the existing ones. I am not going to introduce an exceptional case here but creating our pair-plot grid using Matplotlib.

Plot Grid Area

Initially we need to create a grid plot area using the subplots function of matplotlib like below.

fig, axis = plt.subplots(nrows=3, ncols=3)

For a pair-plot grid you should give the same row and column size because we are going to plot pairwise. Now we can prepare a plot function for the plot grid area we created. If we have 3 features in our dataset as this example we can loop through the features per feature like this:

for i in range(0, 3):
    for j in range(0, 3):
        plotPair()

For cleaner code it is better to move the single pair plotting to another function.

Below is a function I created for one of my master’s degree coursework assignments in December 2021 at the University of London. Plotting a single pair in a grid needs to get the current axis for the current grid cell and identify the current feature data values on the current axis. Another thing to consider is where to render the labels of axes. If we were plotting a single chart it would be easy to render the labels on each axis of the chart. But in a pair plot it is better to plot the labels on the left-most and bottom-most of the grid area so that we won’t bother the inner subplots with the dirty labeling.

def plot_single_pair(ax, feature_ind1, feature_ind2, _X, _y, _features, colormap):
    """Plots single pair of features.

    Parameters
    ----------
    ax : Axes
        matplotlib axis to be plotted
    feature_ind1 : int
        index of first feature to be plotted
    feature_ind2 : int
        index of second feature to be plotted
    _X : numpy.ndarray
        Feature dataset of of shape m x n
    _y : numpy.ndarray
        Target list of shape 1 x n
    _features : list of str
        List of n feature titles
    colormap : dict
        Color map of classes existing in target

    Returns
    -------
    None
    """

    # Plot distribution histogram if the features are the same (diagonal of the pair-plot).
    if feature_ind1 == feature_ind2:
        tdf = pd.DataFrame(_X[:, [feature_ind1]], columns = [_features[feature_ind1]])
        tdf['target'] = _y
        for c in colormap.keys():
            tdf_filtered = tdf.loc[tdf['target']==c]
            ax[feature_ind1, feature_ind2].hist(tdf_filtered[_features[feature_ind1]], color = colormap[c], bins = 30)
    else:
        # other wise plot the pair-wise scatter plot
        tdf = pd.DataFrame(_X[:, [feature_ind1, feature_ind2]], columns = [_features[feature_ind1], _features[feature_ind2]])
        tdf['target'] = _y
        for c in colormap.keys():
            tdf_filtered = tdf.loc[tdf['target']==c]
            ax[feature_ind1, feature_ind2].scatter(x = tdf_filtered[_features[feature_ind2]], y = tdf_filtered[_features[feature_ind1]], color=colormap[c])

    # Print the feature labels only on the left side of the pair-plot figure
    # and bottom side of the pair-plot figure. 
    # Here avoiding printing the labels for inner axis plots.
    if feature_ind1 == len(_features) - 1:
        ax[feature_ind1, feature_ind2].set(xlabel=_features[feature_ind2], ylabel='')
    if feature_ind2 == 0:
        if feature_ind1 == len(_features) - 1:
            ax[feature_ind1, feature_ind2].set(xlabel=_features[feature_ind2], ylabel=_features[feature_ind1])
        else:
            ax[feature_ind1, feature_ind2].set(xlabel='', ylabel=_features[feature_ind1])

Let’s go back to the initial plotting of the grid area and adjust the call of plot_single_pair function. We can adjust the figure size of the grid area using fig.set_size_inches depending on the feature count so that we can prepare a well-scaled area.

colormap={0: "red", 1: "green", 2: "blue"}

fig.set_size_inches(feature_count * 4, feature_count * 4)

# Iterate through features to plot pairwise.
for i in range(0, 3):
    for j in range(0, 3):
        plot_single_pair(axis, i, j, X, y, features, colormap)

plt.show()

In my plot-single_pair function notice that I also used a colormap dictionary. This dictionary is used to color the classes (labels) of the dataset to distinguish in a scatter plot or a histogram and makes it look more beautiful.

Here is my final grid plot function for pair-plot:

def myplotGrid(X, y, features, colormap={0: "red", 1: "green", 2: "blue"}):
    """Plots a pair grid of the given features.

    Parameters
    ----------
    X : numpy.ndarray
        Dataset of shape m x n
    y : numpy.ndarray
        Target list of shape 1 x n
    features : list of str
        List of n feature titles

    Returns
    -------
    None
    """

    feature_count = len(features)
    # Create a matplot subplot area with the size of [feature count x feature count]
    fig, axis = plt.subplots(nrows=feature_count, ncols=feature_count)
    # Setting figure size helps to optimize the figure size according to the feature count.
    fig.set_size_inches(feature_count * 4, feature_count * 4)

    # Iterate through features to plot pairwise.
    for i in range(0, feature_count):
        for j in range(0, feature_count):
            plot_single_pair(axis, i, j, X, y, features, colormap)

    plt.show()

Pair-Plot a Dataset

Now let’s prepare a dataset and plot using our custom pair-plot implementation. Notice that in my plot_single_pair function I passed the feature and target values as the numpy.ndarray type.

Let’s get the iris dataset from the SciKit-Learn dataset collection and do a quick exploratory data analysis.

from sklearn import datasets
iris = datasets.load_iris()

Here are the targets (classes) of the iris dataset:

iris.target_names
array(['setosa', 'versicolor', 'virginica'], dtype='<U10')

And here we can see the feature names and a few lines of the dataset values.

iris_df = pd.DataFrame(iris.data, columns = iris.feature_names)
iris_df.head()

	sepal length (cm)	sepal width (cm)	petal length (cm)	petal width (cm)
0	5.1	3.5	1.4	0.2
1	4.9	3.0	1.4	0.2

Since iris.data and iris.target are already of type numpy.ndarray as I implemented my function I don’t need any further dataset manipulation here. Now let’s finally call myplotGrid function and render the pair-plot for the iris dataset.

Note that you can change color per target in colormap as you wish.

myplotGrid(iris.data, iris.target, iris.feature_names, colormap={0: "red", 1: "green", 2: "blue"})

And here is my custom pair-plot output:

For further research I encourage you to go and do your exploratory data analysis and take a look at the correlation coefficient analysis to get more insights for pair-wise analysis.

End Point Liquid Galaxy at GEOINT Symposium

2017-05-22T00:00:00+00:00

End Point Liquid Galaxy will be coming to San Antonio to participate in GEOINT 2017 Symposium. We are excited to demonstrate our geospatial capabilities on an immersive and panoramic 7 screen Liquid Galaxy system. We will be exhibiting at booth #1012 from June 4-7.

On the Liquid Galaxy, complex data sets can be explored and analyzed in a 3D immersive fly-through environment. Presentations can highlight specific data layers combined with video, 3D models, and browsers for maximum communications efficiency. The end result is a rich, highly immersive, and engaging way to experience your data.

Liquid Galaxy’s extensive capabilities include ArcGIS, Cesium, Google Maps, Google Earth, LIDAR point clouds, realtime data integration, 360 panoramic video, and more. The system always draws huge crowds at conferences; people line up to try out the system for themselves.

End Point has deployed Liquid Galaxy systems around the world. This includes many high profile clients, such as Google, NOAA, CBRE, National Air & Space Museum, Hyundai, and Barclays. Our clients utilize our content management system to create immersive and interactive presentations that tell engaging stories to their users.

GEOINT is hosted and produced by the United States Geospatial Intelligence Foundation (USGIF). It is the nation’s largest gathering of industry, academia, and government to include Defense, Intelligence and Homeland Security communities as well as commercial, Fed/Civil, State and Local geospatial intelligence stakeholders.

We look forward to meeting you at booth #1012 at GEOINT. In the meantime, if you have any questions please visit our Liquid Galaxy website!

Cesium on the Liquid Galaxy

2016-07-13T00:00:00+00:00

Data visualization continues to evolve, with ever-more complex data sets available openly, and a corresponding increased pace in visualization tools. In mapping GIS data, the Cesium app is gaining quite a bit of traction. As we continue to branch out with new functionality and visualization apps for the Liquid Galaxy, we wanted to try the Cesium app as well.

Cesium is written all in JavaScript WebGL and offers some nice advantages over other engines: it’s open source, it’s flexible, and it’s quick. It can accept an array of points, shapefiles, 3D models, and even KML. The JavaScript then chews these up and delivers a nice consistent 3D environment that we can fly through with the SpaceNav controller, set scenes in a presentation to tell a story, or mix together with video or graphic popups for a fully immersive multimedia experience. Cesium is open source, and provides a great deal of flexibility and accessibility to build different kinds of data visualizations and interactions. There are a lot of new startups exploiting this platform and we welcome the opportunity to work with them.

As we’ve written previously, the main advantage of the Liquid Galaxy platform is the ability to adjust the viewing angle on each screen to match the physical offset, avoiding (as much as possible) artificial distortions, fisheye views, or image stretching. The trickiest bit of this project was setting the distributed camera driver, which takes input from the SpaceNav controller and correctly aligns the view position for each of the geometrically offset displays. Once the math is worked out, it’s relatively quick work to put the settings into a rosbridge WebSockets driver. Once again, we’re really enjoying the flexibility that the ROS architecture grants this system.

Looking forward, we anticipate this can open up many more visualizations for the Liquid Galaxy. As we continue to roll out in corporate, educational, and archival environments such as real estate brokerages, hospitality service providers, universities, and museums, the Cesium platform will offer yet another way for our customers to visualize and interact with their data.

How to Add Labels to a Dimple JS Line Chart

2016-05-17T00:00:00+00:00

I was recently working on a project that was using DimpleJS, which the docs describe as “An object-oriented API for business analytics powered by d3”. I was using it to create a variety of graphs, some of which were line graphs. The client had requested that the line graph display the y-value of the line on the graph. This is easily accomplished with bar graphs in Dimple, however, not so easily done with line graphs.

I had spent some time Googling to find what others had done to add this functionality but could not find it anywhere. So, I read the documentation where they add labels to a bar graph, and “tweaked” it like so:

var s = myChart.addSeries(null, dimple.plot.line);
.
.
.
/*Add prices to line chart*/
s.afterDraw = function (shape, data) {
  // Get the shape as a d3 selection
  var s = d3.select(shape);
  var i = 0;
  _.forEach(data.points, function(point) {
    var rect = {
    x: parseFloat(point.x),
    y: parseFloat(point.y)
  };
  // Add a text label for the value
  if(data.markerData[i] != undefined) {
    svg.append("text")
    .attr("x", rect.x)
    .attr("y", rect.y - 10)
    // Centre align
    .style("text-anchor", "middle")
    .style("font-size", "10px")
    .style("font-family", "sans-serif")
    // Format the number
    .text(data.markerData[i].y);
  }
  i++
});

Some styling still needs to be done but you can see that the y-values are now placed on the line graph. We are using lodash on this project but if you do not want to use lodash, just replace the _.forEach (line 10)and this technique should just plug in for you.

If you’re reading this it’s likely you’ve run into the same or similar issue and I hope this helps you!

Intro to DimpleJS, Graphing in 6 Easy Steps

2015-10-15T00:00:00+00:00

Data Visualization is a big topic these days given the giant amount of data being collected. So, graphing this data has been important in order to easily understand what has been collected. Of course there are many tools available for this purpose but one of the more popular choices is D3.

D3 is very versatile but can be a little more complicated than necessary for simple graphing. So, what if you just want to quickly spin up some graphs quickly? I was recently working on a project where we were trying to do just that. That is when I was introduced to DimpleJS.

The advantage of using DimpleJS rather than plain D3 is speed. It allows you to quickly create customizable graphs with your data, gives you easy access to D3 objects, is intuitive to code, and I’ve found the creator John Kiernander to be very responsive on Stack Overflow when I ran in to issues.

I was really impressed with how flexible DimpleJS is. You can make a very large variety of graphs quickly and easily. You can update the labels on the graph and on the axes, you can create your own tooltips, add colors and animations, etc..

I thought I’d make a quick example to show just how easy it is to start graphing.

Step 1

After including Dimple in your project, you simply create a div and give it an id to be used in your JavaScript.

<head>
  <script src="http://d3js.org/d3.v3.min.js"></script>
  <script src="http://dimplejs.org/dist/dimple.v2.0.0.min.js"></script>
</head>
<body>
  <div id="chart" style="background:grey"></div>
</body>

Step 2

In your JavaScript you use Dimple to create an SVG, targeting the element with your id, in this case “chart” and I’ve given it some size options.

var svg1 = dimple.newSvg("#chart", 600, 400);

Step 3

Then, you get your data set through an API call, hardcoded, etc.. I’ve hardcoded some sample data of weeks and miles ran for two runners.

var data1 = [
    [
      {week: 'week 1', miles: 1},
      {week: 'week 2', miles: 2},
      {week: 'week 3', miles: 3},
      {week: 'week 4', miles: 4}
    ],
    [
      {week: 'week 1', miles: 2},
      {week: 'week 2', miles: 4},
      {week: 'week 3', miles: 6},
      {week: 'week 4', miles: 8}
    ]
  ];

Step 4

Next, you set up your axes. You can create multiple x and y axes with Dimple but for the sake of this example I will just create one x and one y. The two that I use most are category and measure. Category is used for string values, here the runners’ names. One word of caution form the docs for the measure axes is that “If the field is numerical the values will be aggregated, otherwise a distinct count of values will be used.”. This means that if I had two runners who were both named Bill in the same data set, and say the first Bill ran 1 mile in week 1 and the second Bill ran 2 miles in week 1, Dimple would graph one x-value Bill and one y-value 3 rather than two Bills, one with 1 mile and one with 2.

var xAxis = chart1.addCategoryAxis("x", "week");
var yAxis = chart1.addMeasureAxis("y", "miles");

Step 5

Now we’ve set up the axes, so we want to actually plot the points. In Dimple we call this “adding a series”. We are telling it what kind of graph we want, and what to call the data (in this case, the runner’s name), and we assign it the data we’d like to use.

s1 = chart1.addSeries("Bill", dimple.plot.line);
  s1.data = data1[0];
  s1.plot=dimple.plot.line;

  s2 = chart1.addSeries("Sarah", dimple.plot.line);
  s2.data = data1[1];
  s2.plot=dimple.plot.line;

Step 6

Lastly, we simply tell Dimple to actually draw it.

chart1.draw(1000);

The code in full as well as a working JSBin can be found here.

var chart1 = new dimple.chart(svg1);
var xAxis = chart1.addCategoryAxis("x", "week");
var yAxis = chart1.addMeasureAxis("y", "miles");

var data1 = [
  [
    {week: 'week 1', miles: 1},
    {week: 'week 2', miles: 2},
    {week: 'week 3', miles: 3},
    {week: 'week 4', miles: 4}
  ],
  [
    {week: 'week 1', miles: 2},
    {week: 'week 2', miles: 4},
    {week: 'week 3', miles: 6},
    {week: 'week 4', miles: 8}
  ]
];

s1 = chart1.addSeries("Bill", dimple.plot.line);
s1.data = data1[0];
s1.plot=dimple.plot.line;
s2 = chart1.addSeries("Sarah", dimple.plot.line);
s2.data = data1[1];
s2.plot=dimple.plot.line;
chart1.draw(1000);

Graphing System Statistics with Grafana

2015-08-21T00:00:00+00:00

Graphing System Statistics (the old fashioned way)

Since the mid 2000s system administrators who wanted to have a visual representations of their systems statistics had access to Graphite. This tool allows for elaborating graphs of values collected periodically to provide a representation of the data over time. Coupling this feature with collectd, which among many built-in supported metrics offer the possibility of sending system statistics to a central location running Graphite, allows to create a single portal for viewing statistics of your entire infrastructure easily. While this still remains a nice setup the graphical visualization capabilities of Graphite and rrdtool left some room for growth.

Enter Grafana

Grafana is a Graphite installation front-end that offers a very robust graphing/plotting library (provided by Flot) along with templates for creating similar displays for multiple datasets. Here you can see a comparison of the same dataset in both graphing libraries:

Graphite (rrdtool)

Grafana (Flot)

Data Analysis

Once you have setup collectd and Graphite to gather simple metrics from the servers you wish to monitor, you will easily have some basic instrumentation for focusing on system resources monitoring which is very helpful to keep an eye on your infrastructure performance and health. However the more data you start to harvest with collectd’s many plugins the bigger will be the need for some aggregation and analysis of the data to better understand what the data could be communicating. In this example there are two graphs, the top is a measure of the network traffic going across the external interface of a network firewall, and the bottom is a measure of the total traffic transformed using a logarithmic base 10 function on the data.

Within the logarithmic graph it’s easier to perceive the magnitude of the value, as a change in that graph of 1.0 in either direction would reflect a 10 fold change in the underlying data. Using this approach gives an operator a view of magnitude of the change and would so being able to easily track any spikes in the data values. Luckily Graphite offers a huge number of possible functions to elaborate the data before actually displaying it in the graph, they are all clearly documented here.

Dashboards

Going further you may want different contextual groups to aggregate systems by host or by application and with Grafana you can create a dashboard view which is customizable and can be populate with all the needed data. Here is an example of a dashboard for a system we have already seen:

Mousing over any of the data within the charts will allow for detailed examination of the data values measured at that time period and provides the legend for each color in the chart. Changing the dataset timeframe is as simple as adjusting the dropdown near the top of the page, or clicking and dragging a duration onto one of the graphs.

The graphing library Flot provides a huge number of features and a very modern visual style which improved on what Graphite had to offer with rrdtool.

Conclusions

Graphite and collectd offered (and still do) a really robust data collection and analysis tools for measuring groups of computers. However this data seemed trapped in a display front-end which just could not meet the needs administrators who wanted to investigate deeper the collected data. Now Grafana provides a vastly improved graphing engine (also thanks to Flot), and combines all the needed tools like templates and dashboards to really empower users and system administrators about what they could do with the collected data.

We won’t be the first to say it but we do confirm that the combination of the great gathering and analysis capabilities powered by Graphite and collectd with a robust front-end with Grafana creates a very powerful tuning and monitoring stack.

GIS Visualizations on the Liquid Galaxy

2014-03-25T00:00:00+00:00

The Liquid Galaxy presents an incredible opportunity to view Google Earth and Google Street View, but did you know that the platform is also amaza-crazy good at visualizing GIS data?

Geographic Information Systems are concerned with collecting, storing, and manipulating data sets that include geographic coordinates, and with displaying this raw and analyzed data on maps. In the 2D world this usually involves colored pencils or highlighted polygons. As computing power advances many GIS consultancies have extended the GIS visualization methods to see complex 3D visualizations on digital maps. The Liquid Galaxy takes this concept forward another step: see your data across an immense landscape of pixels in a geometrically-adjusted immersive world.

The Liquid Galaxy has separate instances of Google Earth running on each screen. In a standard Liquid Galaxy this means that each 1080x1920 screen is getting a full resolution image. It also means that the viewing angle for each screen matches the physical angle of that screen. Together, those elements can show geographical information at an incredible scale and resolution.

End Point has developed skills and methods to take GIS data sets (commonly found as KML files, but any spreadsheet of data with latitude and longitude can be adapted), apply the geometric adjustments for multiple screens, and build incredible presentations for our clients. Sample data (in the video) includes: utility power usage, offshore oil & gas leases, population growth by state, school districts, and earthquakes along the San Andreas fault.

End Point welcomes the opportunity to work with GIS consultancies in bringing their data to the Liquid Galaxy and then presenting an immersive visual platform to their clients and customers. For more information, please see our Liquid Galaxy website.