End Point’s CasePointer business acquired by Deloitte

2024-06-03T00:00:00+00:00

We are proud to announce that Deloitte has acquired the industry-leading public health disease surveillance business, CasePointer, from End Point Corporation as of June 1, 2024. This acquisition brings to Deloitte, End Point’s experienced and skilled software engineers, cloud developers, DevOps engineers, epidemiologists, and former public health officials.

Since 2008, End Point’s CasePointer business has worked with open-source public health software to modernize tools, automate processes, improve the user experience, and provide better return on investment to state and local public health agencies for their disease surveillance and epidemiologic needs.

Jon Jensen, End Point’s former chief technical officer, has joined Deloitte with this transaction. He said: “We are excited to work with Deloitte’s team of professionals and serve their broad client base. Together, we expect to make an even greater impact for government health organizations.”

Deloitte is only acquiring assets, clients, and personnel related to End Point’s CasePointer business. All other business lines remain with End Point, including systems administration, database engineering, e-commerce programming, application development, and our immersive GIS platform, VisionPort.

See Deloitte’s Public Health Transformation website for further information.

Achieving Healthcare Data Interoperability

2023-07-28T00:00:00+00:00

(Co-authored by Jon Jensen.)

Background

For many years, healthcare providers, public health experts, and software developers have been hard at work on standards and methods for interconnecting systems to share vast and ever-expanding patient and lab data.

In the late 1980s the non-profit organization Health Level Seven International released the HL7 data standard. That was updated many times up to the year 2007 with HL7 v2.5.1 which is still widely used today.

New healthcare data standards have emerged, including HL7 v3, the Consolidated Clinical Document Architecture (C-CDA), the Fast Healthcare Interoperability Resource (FHIR, pronounced “fire”), and more. These are not yet as broadly adopted as HL7 v2.5.1.

Under the Health Insurance Portability and Accountability Act of 1996 (HIPAA), most of this data is classified as PII (personally identifiable information) and/or PHI (personal health information) and must be kept secure and protected at all times, so that it will not be accessed without proper authorization or patient consent.

The data dilemma

Healthcare data systems now routinely communicate in many ways, but there are sometimes incompatibilities when patient data travels from one system to another.

The U.S. Centers for Disease Control (CDC) encourages and sometimes incentivizes, but does not mandate, adoption and implementation of common standards. So each jurisdiction’s public health agency (PHA), lab, and vendor may be using different and only partially overlapping standards.

Different labs, doctors, and public health workers use varying conventions for things like facility names and lab codes, which presents problems for interpreting and reporting data. For example, “KU MAIN” may be recognized as the main campus of the University of Kansas Medical Center in nearby communities or states, while providers elsewhere may not recognize this code, or may mistake it for a similarly named facility in their area. The University of Kansas Health System consists of a large network of satellite facilities that span the states of Kansas and Missouri, rather than just one location in a single state. This may create additional confusion about which state jurisdictions should be involved.

This semantic category is called a vocabulary, and if two systems use incompatible vocabulary, one of them, or an intermediary, must have a mapping from the terms in one vocabulary to another.

The syntactic category of challenge is called a grammar, and covers the kinds of message types and actions a system works with.

An article published in December 2022 by Dr. Naheed Ali identifies “inconsistent data and lack of standardized data structure” as the number one issue to be overcome regarding the Electronic Health Record (EHR) interoperability challenges facing healthcare professions today:

One of the biggest EHR interoperability challenges in healthcare interoperability is managing inconsistent data from multiple sources. Information stored in different databases can have a variety of formats and data types that are not easily compatible with one another. A single record may contain different information about a patient’s medical history or treatment plan, making it more difficult for different systems to interpret correctly. —Dr. Naheed Ali, EHR Interoperability Challenges and Solutions

Cooperating for a healthier future

Although great strides have been made to digitize healthcare data and improve interoperability, much remains to be done. Newer standards (like HL7 FHIR, released in 2014) face an uphill battle for industry adoption to become as widely utilized as HL7 v2.5.1 (from 2007) and C-CDA (from 2011).

While networks of hospitals and laboratories typically set up EHR platforms built by companies like Cerner or Epic to manage patient data, public health agencies require additional applications to aid in the identification, tracking, and reporting of reportable health conditions from those patient EHRs.

Public health services in the United States are provided by various federal, state, tribal, territorial, regional, municipal, and other bodies that decide which software they will use to support their communities. Some look to software offered by the CDC or software firms to provide the application functions they require. Others contract with consulting firms to develop custom software. Still others opt to develop and maintain everything in house.

Getting so many diverse software systems to work together is difficult due to a phenomenon known as the quadratic cost problem. This occurs when the need for bi-directional connections between software applications increases exponentially, instead of linearly, with each component added to a system.

An organization called TechChange explains this concept and more in their video Standards and Interoperability in Digital Health: Explained.

The book Patient Safety: Achieving a New Standard for Care identifies in chapter 4 this concern:

The fact that there is no standard means of representing the data for any of these datasets or requirements is astonishing and highlights the amount of unnecessary work performed by health care and regulatory organizations to prepare, transmit, and use what amount to custom reports.

Nearly 20 years after those words were written, and almost a decade after the FHIR standard was introduced, the variety of standards and systems has not narrowed. Change is difficult.

Health Information Exchanges

One architectural attempt to simplify data sharing is by connecting applications of each jurisdictional segment within a larger governing body to a centralized Health Information Exchange (HIE).

An example of this would be an individual state public health organization connecting its own software to an HIE, along with the software currently in use by each local jurisdiction within that state. Each jurisdiction does not have to use the same software, but can communicate by using its connection to the HIE rather than individual bi-directional connections to every other application used throughout the state.

This sounds like just what is needed, and likely is for some use cases.

But HIEs generally work at the syntactic (grammar) level. Many HIEs have agreements with the facilities they serve, that they will not change the content of messages in any way. Unfortunately that means most HIEs essentially work as a passthrough.

They thus are helpless when it comes to the wide variety of nonstandard content semantics that they encounter, and have no way of correcting vocabularies, so they pass on junk data.

Even a small, steady volume of unusable messages can at best clog up manual review queues and overwhelm already overworked staff, and at worst give a false sense that all data are being ingested and reported on, when they are not.

Something like an HIE that is capable of semantic (vocabulary) inspection, translation, and rule-based routing, customizable per reporter for the jurisdiction’s changing needs, is called for.

The CasePointer software suite

We at End Point are solving the interoperability problem for our client jurisdictions’ public health disease surveillance with CasePointer, a new suite of proven open source software. It interoperates with many other systems, including HIEs, and is configurable to map varying local data conventions into a single system that routes messages and reporting as needed.

EpiTrax disease surveillance system

We began in 2008 to work with software for public health using the open-source TriSano disease surveillance system.

In 2017, the Utah Department of Health and Human Services (DHHS) released a new open source disease surveillance application called EpiTrax to replace their use of TriSano. EpiTrax is a more capable, comprehensive, and supportable application that supports all 170+ reportable health conditions identified by the National Notifiable Disease Surveillance System (NNDSS).

As open source software, we and others can use it, extend its capabilities, and contribute back to a shared source code repository.

Today EpiTrax serves as the central application within the CasePointer suite.

EMSA preprocessing powerhouse

The separate application Electronic Message Staging Area (EMSA) was also designed by the team at Utah DHHS to work in tandem with EpiTrax. EMSA ingests electronic case and lab reports (eCRs & ELRs) from various reporters and allows public health organizations to create extremely powerful custom configurations capable of message inspection, semantic (vocabulary) mapping and transformation, and routing, tailored to each jurisdiction’s needs and preferences.

Once these EMSA rules are set up, the healthcare data can flow continuously while automatically being pre-processed by EMSA according to its matching jurisdiction with minimal manual intervention by the area’s public health agency. PHA staff can focus on their primary objective: investigating and tracking disease to protect the health of their community.

Investigators can customize their individual dashboards within EpiTrax to analyze, sort, trace, and process the data prepared by EMSA in the way that is most efficient for them. Working together, these two powerhouse applications provide the cost and time saving benefits that public health organizations around the country need.

NMI automatic CDC reporting

Once a public health case reaches a certain status within EpiTrax, certain data must be reported per current requirements of the CDC. Historically, this has been a manual and tedious process for investigators involving referencing a text file with the latest CDC guidelines as they process these reports.

More recently, the NNDSS Modernization Initiative (NMI) by the CDC has provided options for a much more automated method of reporting the necessary case data. This is a critical part of the process for PHAs, as they receive funding for their organization based on their ability to meet CDC reporting requirements and interoperability standards.

The NMI module is integrated into the CasePointer workflow to ensure that once data reporting has been configured for a case, any updates will be automatically sent to the CDC without additional manual processing.

CasePointer Disease Reporting Portal

During COVID-19, nontraditional reporters such as pharmacies, university health centers, community drive-up testing centers, and others not part of our state clients’ formal public health infrastructure, needed to be able to submit test results to the state.

We developed a specialized disease reporting portal, initially focusing on COVID-19, to accept simple batch uploads of test results and, based on changing policy, to forward test data to the disease surveillance system. This separate system had no access to the main EpiTrax disease surveillance system, or to any other reporters’ data.

Bringing it all together

CasePointer provides two prominent strengths:

First, its ability to map the endless local conventions used by those reporting the data into a unified standard for the rest of the workflow.
Second, the customizability and agility of the CasePointer applications are unique among available disease surveillance solutions.

CasePointer puts the ability to customize data collection and surveillance in the hands of the public health experts. New fields can be added into an investigation form with the click of a button. The ability to quickly pivot and document disease information is critical in a quickly changing public health landscape.

Join the evolution of public health technology

Such highly configurable, unified, robust tools can be a game changer for public health jurisdictions in need of an affordable, extensible solution that meet all their needs. CasePointer can even work to further streamline systems that are already integrated with an HIE.

Within the United States, four states have already partnered with us to implement EpiTrax. We are helping them reduce unnecessary manual data processing costs, more effectively manage caseloads, and further automate their data workflows.

If this piqued your curiosity, contact us to learn more about CasePointer and how our team is evolving public health support with customized data solutions.

We would love to support your efforts to protect the well-being of people and communities you care about!

CSTE Conference 2023 Retrospective

2023-07-14T00:00:00+00:00

End Point is proud to have been an exhibitor at the 2023 Council of State and Territorial Epidemiologists (CSTE) conference in Salt Lake City, Utah.

We are grateful to have had the opportunity to showcase our public health software suite offering, CasePointer, to numerous state, tribal, territorial, and international epidemiologists.

The conference served as a meeting place to hear about the latest research in the fields of informatics and disease surveillance, and discuss the future of data management on state and national levels, with many talks exploring the technologies that will become future standards and practices.

The End Point booth was abuzz with visitors getting hands-on demos using the EpiTrax software as well as learning about the other aspects of the CasePointer suite, such as the Electronic Message Staging Area (EMSA), our new disease reporting portal, and the EpiTrax NMI module.

The CasePointer demo highlighted its capability of streamlining the management of epidemiological data with a user-friendly and highly customizable interface.

The CSTE conference provides a valuable opportunity to engage with epidemiologists and gain insights into the daily obstacles they encounter in data reception, management, and sharing. It offers a platform to learn about their strategies for overcoming these challenges within their respective jurisdictions.

One of the common topics that came up was tackling the challenge of file transfers using current protocols and systems in place. We were happy to share our successes using EMSA to customize how data is received and uploaded to EpiTrax, making transfers faster, more reliable, and easier to manage.

We look forward to further conversations with the visitors that stopped by our booth, and hope that you enjoyed the locally crafted chocolates and caramels and saved some for the trip home!

Are you interested in a demo of the CasePointer suite? Reach out to us at ask@endpointdev.com to set up a meeting.

Announcing CasePointer

2023-06-20T00:00:00+00:00

Photo by Diva Plavalaguna, Pexels license

End Point is pleased to announce CasePointer, the expanded branding of our line of public health systems and services. Before this we referred to it simply by the name of one of its components, EpiTrax.

The name CasePointer more appropriately reflects the broad scope of services we provide within the disease reporting and surveillance sector.

This change coincides with the launching of the dedicated CasePointer website at www.casepointer.com and our participation next week in this year’s conference of CSTE (Council of State and Territorial Epidemiologists) in Salt Lake City, Utah. We will be staffing a booth there on the exhibition floor. Come visit us!

Starting in 2008, End Point helped develop and maintain the TriSano open source disease surveillance system. In 2019 we migrated our customer, the Kansas Department of Health and Environment, from TriSano to a new disease surveillance system called EpiTrax, which was developed by the state of Utah, and is also open source.

This timeline shows those and other key points on our journey:

Due to the need for major additional features and capabilities, especially since the onset of the COVID-19 pandemic, End Point is now integrally involved with public health agencies in four states. For each of them we configured, deployed, maintain, and enhance the cloud hosting, relevant software, and databases to run those components they need, including:

the EpiTrax disease surveillance system
EMSA (the Electronic Message Staging Area)
the NMI (NNDSS Modernization Initiative) module
End Point’s Disease Reporting Portal
End Point’s merge tool
Mirth Connect

All of these interoperate harmoniously around the central EpiTrax application and database.

Our team is composed of public health professionals, software engineers, database specialists, and security experts. We are committed to our public health clients and their needs. We are here to help you help our communities and look forward to discussing the CasePointer approach with you and your team.

eCR Coordinator job opening

2023-05-23T00:00:00+00:00

We are seeking a full-time Electronic Case Reporting (eCR) Coordinator to provide high-level expertise in the validation and automation of eCR messages.

This position is responsible for establishing complete, up-to-date, timely, and automated reporting of individual-level data, and automation of incoming EHR data. Position is remote, in the United States only.

The person in this role will be responsible for coordinating and overseeing the eCR onboarding process for our clients. Additionally, the person will become our subject matter expert for the NNDSS Modernization Initiative (NMI) and how systems in place within EpiTrax can be configured to report through the NMI module.

Job responsibilities

eCR onboarding—40%
- Assist the on-boarding team with eCR reporters.
- Develop Structure, Dictionary, LOINC and SNOMED codes in EMSA.
- Assist in eCR message structure approval.
- Assist with EMSA management.
Troubleshooting—30%
- Issues found by the program epis and the LHDs.
- Issues found by the surveillance team.
- Create trouble tickets for issues.
Testing—15%
- Test the structure for incoming messages.
- Test new enhancement functionality.
- Test bug fixes.
Grant document development—5%
- Assist with determining next steps in development to be included in grants.
- Develop documentation for grants.
Process documentation—5%
- Thoroughly document changes and the processes put in place.
NMI—5%
- Work with state and CDC teams to determine MMG onboarding schedule.
- Map EpiTrax variables to PHIN VAD variables in the NMI module.
- Document all mappings and changes.

Skills required

Working knowledge of public health surveillance or EHR and healthcare data
Good communication skills, both spoken and written
Ability to analyze problems
Good testing skills
Ability to multitask and set priorities

Skills to have, or to learn on the job

In-depth knowledge of eCR messaging
In-depth knowledge of EMSA
In-depth knowledge of EpiTrax
Working knowledge of NMI

For some of our clients you will need to submit to and pass a criminal background check.

What work here offers:

Starting annual salary range: $50k–$85k
Paid holidays (10 per year) and vacation (2 weeks per year)
Health insurance subsidy and 401(k) retirement savings plan
Collaboration with knowledgeable, friendly, helpful, and diligent co-workers around the world
Freedom from being tied to one office location
Flexible, sane work hours
Annual bonus opportunity

Get in touch with us:

~~Please email us an introduction to jobs@endpointdev.com to apply.~~ (This job has been filled.)

Include your location, a resume, your LinkedIn URL if you have one, and whatever else may help us get to know you.

We look forward to hearing from you! Direct work seekers only, please—this role is not for agencies or subcontractors.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of sex/gender, race, religion, color, national origin, sexual orientation, age, marital status, veteran status, or disability status.

How To Set Up pgTAP for Writing PostgreSQL Database Unit Tests

2023-04-12T00:00:00+00:00

In a previous blog post, my colleague Josh Tolley introduced the concept of using pgTAP, a set of Postgres functions designed for writing unit tests within the database. This post will serve as a supplement to Josh’s post, focusing on the process of setting up pgTAP for testing a Postgres database.

Installing & setting up pgTAP

Note: make sure you have Postgres installed on your system. If you don’t have it, you can follow the Postgres documentation to learn how to install it.

To install pgTAP for Postgres, you will need to follow these steps:

Download the pgTAP source code from its GitHub repository.
```
git clone https://github.com/pgtap/pgtap.git
```
Navigate to the pgtap directory.
```
cd pgtap
```
Install with make.
```
make && make install
```
Finally, connect to your Postgres database using psql or any other Postgres client and run the following SQL command to create the pgTAP extension.
```
CREATE EXTENSION pgtap;
```

Now you should have pgTAP installed and ready to use in your Postgres database. If you face issues with installation, visit the pgTAP documentation for further help.

Writing simple database tests with pgTAP

Test to check if a table exists:

SELECT plan(1);
SELECT has_table('public', 'your_table', 'Table your_table should exist');
SELECT finish();

Test to check if a column exists in a table:

SELECT plan(1);
SELECT has_column('public', 'your_table', 'your_column', 'Column your_column should exist in your_table');
SELECT finish();

Test to check if a function returns the expected result:

SELECT plan(1);
SELECT is(your_function(), expected_result, 'your_function() should return expected_result');
SELECT finish();

More pgTAP tests

Test to check if a trigger is triggered

Consider the following example. We have a table named employees and a table named audit_log. When a new employee is added to the employees table, a trigger named insert_employee_trigger fires and logs the new employee’s ID and creation timestamp in the audit_log table. Let’s test if the trigger gets fired and produces the expected outcome.

Create the tables.

CREATE TABLE employees (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL
);

CREATE TABLE audit_log (
  id SERIAL PRIMARY KEY,
  employee_id INTEGER NOT NULL,
  created_at TIMESTAMP NOT NULL
);

Create the trigger and trigger function.

CREATE OR REPLACE FUNCTION insert_employee_trigger_function()
RETURNS TRIGGER AS $$
BEGIN
  INSERT INTO audit_log (employee_id, created_at) VALUES (NEW.id, NOW());
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER insert_employee_trigger
  AFTER INSERT ON employees
  FOR EACH ROW
  EXECUTE FUNCTION insert_employee_trigger_function();

Now, write a test using pgTAP to test the trigger.

-- Set the test plan
SELECT plan(1);

-- Prepare test data and perform operations that should fire the trigger
INSERT INTO employees (name) VALUES ('John Doe');


-- Check if the desired outcome of the trigger has occurred
SELECT is(
  (SELECT name FROM employees WHERE id = (SELECT employee_id FROM audit_log ORDER BY id DESC LIMIT 1)),
  'John Doe',
  'The insert_employee_trigger should have been fired and produced the expected outcome'
);

Test to check if a constraint is enforced

For this example, we’ll use the following two tables:

customers table with columns id and name.
orders table with columns id, customer_id, and amount
- There is a foreign key constraint between the orders table and the customers table on the customer_id column.

First, create the two tables and add the foreign key constraint.

CREATE TABLE customers (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL
);

CREATE TABLE orders (
  id SERIAL PRIMARY KEY,
  customer_id INTEGER REFERENCES customers(id),
  amount INTEGER
);

Now, create a test script named fk_test.sql with the following content.

BEGIN;

-- Create test data in customers table
INSERT INTO customers (id, name) VALUES (1, 'Customer 1');

-- Declare the number of tests
SELECT plan(1);

-- Attempt to insert an order with an invalid customer_id
-- This should fail due to the foreign key constraint
SELECT throws_ok(
    'INSERT INTO orders (id, customer_id, amount) VALUES (1, 969, 100);',
    '23503',
    'insert or update on table "orders" violates foreign key constraint "orders_customer_id_fkey"'
);

-- Conclude the test and discard changes
SELECT finish();
ROLLBACK;

Organizing and naming your test files based on Test Anything Protocol (TAP)

Test Anything Protocol (TAP) is a convention for naming test files and organizing them within a project directory structure. The naming convention involves adding a numeric prefix to the test file name to indicate the order in which the tests should be run. For example, 00-test.sql is the first test file to be executed, followed by 01-another-test.sql, and so on.

The project directory structure usually includes a directory called t (short for “test”) under the project root directory. This t directory contains all the test files following the TAP naming convention.

Using this structure makes it easier to maintain and organize tests, as well as to run them in the correct order. This is especially important when tests depend on the results of previous tests, or when you want to ensure that tests are executed in a specific sequence.

Here’s an example of a project directory structure using the TAP-related naming convention:

my_project/
│
├─ src/
│   ├── function1.sql
│   ├── function2.sql
│   └── function3.sql
│
└─ t/
    ├── 00-test.sql
    ├── 01-another-test.sql
    └── 02-yet-another-test.sql

In this example, the project contains a src directory for the source code (database functions) and a t directory for the test files following the TAP naming convention.

Running your tests using pg_prove

Now that we have written some tests using pgTAP, let’s see how we can run them. We’ll use pg_prove, a utility included with pgTAP, to automate the process and get a summary of the results.

To run your tests with pg_prove, you’ll need to specify the name of the database you want to connect to and the path to your test file. For example, if your test file is located at t/00-test.sql and your database is named epitrax_db, you can run the following command:

pg_prove -d epitrax_db t/00-test.sql

When you run this command, pg_prove will execute all the tests in the specified file and print a summary of the results. The output will include the number of tests run, the number of tests passed, the number of tests failed, and any diagnostic messages or errors that were encountered.

Here’s an example of what the output might look like:

t/00-test.sql .. ok
All tests successful.
Files=1, Tests=5,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.03 CPU)
Result: PASS

In this example, pg_prove ran one test file (00-test.sql) and executed 5 tests. All of the tests passed, so the output shows that the result was PASS.

If any of the tests fail, pg_prove will print a detailed report of the failures, including any error messages or diagnostic information. This can help you quickly identify and fix any issues in your code.

Using pg_prove to automate your tests can save you a lot of time and effort, especially if you have a large number of tests to run. It also makes it easy to integrate your tests into your build process or continuous integration pipeline.

pgTAP in EpiTrax

EpiTrax relies on Postgres and its functions, making pgTAP an ideal tool for testing the platform’s database functions, ensuring data integrity, and maintaining the robustness of the system. By utilizing pgTAP, EpiTrax efficiently validates various aspects of its database structure and behavior.

Conclusion

pgTAP database unit tests provide numerous benefits, including ensuring code quality and simplifying the testing process. This guide should serve as a starting point for setting up pgTAP, and we encourage you to explore the pgTAP documentation and read the aforementioned blog post on automated database testing with pgTAP for more in-depth information on writing and running tests with pgTAP. By integrating pgTAP in projects like EpiTrax, it’s possible to improve overall database performance, stability, and security.

Identifying Vulnerabilities in Code Using Horusec

2023-03-28T00:00:00+00:00

Horusec is an open source tool which, by orchestrating other security tools, identifies security flaws and vulnerabilities in source code. It puts all the possible vulnerabilities it finds into a database for analysis.

Currently, Horusec supports C#, Java, Kotlin, Python, Ruby, Go, JavaScript, TypeScript, PHP, Swift, C, Dart, Elixir, shell, Terraform, Kubernetes, nginx, HTML, and JSON. You can see an up-to-date list of supported languages in Horusec’s docs.

It can also be integrated with CI/CD to execute the scan every time a developer creates a pull request or merge request.

Horusec CLI Installation

Requirements: Docker, Git.

The easiest installation method listed in the docs is curling Horusec’s install script and piping it into bash:

curl -fsSL https://raw.githubusercontent.com/ZupIT/horusec/main/deployments/scripts/install.sh | bash -s latest

Be aware that there is risk to piping unseen commands into the shell like this: It can lead to unintended consequences and it is a bad security practice.

If a user blindly pipes the output of a website response to be run by a shell without fully understanding what each command does, they may inadvertently execute malicious code or perform actions that could compromise the security of the system.

To mitigate these risks, one solution is to use a virtual machine specifically set up for code scanning. This virtual machine should have no production data, no secret keys, and no other sensitive information. It should be used only for the purpose of scanning source code and nothing else.

To instead download an executable directly, as well as for instructions for different platforms, see the installation docs.

VS Code Extension

Horusec has a VS Code extension which is helpful for making complete code analysis with a single click.

Usage

Navigate to the application directory and run the generate command to make a configuration file called horusec-config.json which has a set of customisations:

cd /path/to/project
horusec generate

The start command executes the code scan throughout the repository, searching for possible vulnerabilities:

horusec start -p . --output-format=json --json-output-file=<filename>.json

All the detected vulnerabilities are stored in the named JSON output file.

Classification of Vulnerabilities

Horusec may identify (or, as the docs say, accuse) possible vulnerabilities that aren’t vulnerabilities at all. Possible vulnerabilites need to be classified to sort out those that are wrong.

Here are the classification types from Horusec’s docs:

False positive: Vulnerability found is wrong, because it is accused in a test file or it is not a vulnerability in fact, and is safe code.
Accepted Risk: Vulnerability that was accused, but at the moment, you don’t have the option to correct it, so it is classified as an accepted risk to not raise alarms in future runs.
Corrected: Vulnerability that doesn’t exist and can be considered as corrected.
Vulnerability: A possible security problem found and accused by the analysis.

The hashes to be ignored can be added to the configuration file, horusec-config.json:

{
  ...
  "horusecCliFalsePositiveHashes": [
      "f9e5abe187ad4246daa4e9113e0a11a175347e793fbc40acd7663df67b2f89d2",
      "878c35116d043311403a0a2e8a64f2c8d00479a1c23373dcd50372ff35e123c8"
  ],
  "horusecCliRiskAcceptHashes": [
      "5c9af42834ca77233a0e7afc98df317fb0e6041ea69a109754f278331039f844",
      "b494003277bcd7792390f032ba39e9a860da66ddb1ccfcd8a581978eab744561"
  ],
  ...
}

Ignore Files

To entirely ignore a file or certain paths under the directory, add the paths to the configuration file:

{
  ...
  "horusecCliFilesOrPathsToIgnore": [
      "**/tmp/**",
      "**/.vscode/**"
  ],
  ...
}

Code Scanning in EpiTrax

The use of a code scan tool has become an integral part of our work with the EpiTrax disease surveillance system for public health.

By integrating Horusec into our software development process, we have been able to identify potential security vulnerabilities, code quality issues, and other problems before they can cause any significant harm.

Moreover, it has helped us reduce the overall cost and time associated with bug-fixing and maintenance tasks, making our projects more efficient and effective. By automating the scanning process, we can identify problems quickly and accurately, allowing us to prioritize and address the most critical ones first.

By leveraging the power of code scans, we can ensure that our code is of the highest quality, and our projects are as secure and reliable as possible.

Special thanks to my colleagues Kürşat Kutlu Aydemir and Edgar Mlowe for their insights and feedback on this blog post.

Data Migration Tips

2023-02-04T00:00:00+00:00

When you’re in the business of selling software to people, you tend to get a few chances to migrate data from their legacy software to your shiny new system. Most recently for me that has involved public health data exported from legacy disease surveillance systems into PostgreSQL databases for use by the open source EpiTrax system and its companion EMSA.

We have collected a few tips that may help you learn from our successes, as well as our ~~mistakes~~particularly educational experiences.

Customer Management

Your job is to satisfy your customers, and your customers want to know how the migration is progressing. Give them an answer, even if it’s just a generalization. This may be a burndown chart, a calculated percentage, a nifty graphic, or whatever, but something your project managers can show to their managers, to know more or less how far along things are.

Your job is also to know your system; that’s not the customer’s job. They shouldn’t have to get their data into a specific format for you to make use of it. Be as flexible as possible in the data format and structure you’ll accept. In theory, so long as your customer can provide the legacy data in a machine-readable format, you should be able to use it. Let them focus on getting the data out of their legacy system — which is sometimes quite an effort in itself! Real data is almost always messy data, and your customer will probably want to take the opportunity to clean things up; make it as easy as possible for them to do that, while still ensuring the migration proceeds quickly and smoothly.

Be careful about the vocabulary you use with your customer. Your system and the legacy system probably deal with the same kinds of data, and do generally the same kinds of things. Of course, your software does it better than the old ’n’ busted mess you’re replacing, but in order to be better, your software has to be different from what it’s replacing. Different software means different methods, different processes, and different concepts. You and your customer might use the same words to mean totally different things, and unless you’re careful, those differences can remain hidden well into the implementation process, coming to light just in time to highlight some horrible mistake you’ve made. Try to avoid that. Talk with your customer to make sure your team and their team share a common vocabulary. If your system’s core function is to track widgets, and their legacy system’s function is also to track widgets, make sure you understand the differences between your software’s concept of a widget and their legacy system’s concept of a widget.

Migration Scripting

For our products and our customers, we’ve found pretty much every migration process is different, and each one is a custom programming job. Your team should decide at the beginning what steps the migration needs to include, and what technologies it will use in each step. Here at End Point, we often like to work directly in the database, in SQL. Migrations are all about manipulating data, and SQL is well suited for that task. With whatever technology you use, decide how you’ll use it, to manage the considerations given here. You can change your mind later and refactor accordingly, but always have a plan you’re following.

Design the migration as a sequence of processes. That is, first you might import one type of record, next another type of record that depends on the previous one, followed by several further steps to import data from a third source, clean it, map values from the legacy system to the new system, validate the results, and create records in the destination database. Of course the steps will vary from project to project, but the point is your migration will probably include several steps which need to be run in a specific order, so plan your development conventions accordingly. At End Point, we often like to put each step in a SQL file, and name each file beginning with a number, so you can run each script in order sorted by filename, and achieve the correct result. We might have files called:

01_import_products.sql
02_import_customers.sql
03_import_order_history.sql

It’s also common to implement each step one at a time, and to need to run each step several times as it’s being developed. We find it very helpful to wrap each step in a transaction. Often that means each SQL file begins with a BEGIN; statement, and ends with COMMIT;. Often I’ll leave out the COMMIT until I’m finished working on a file. That way to work on the code I can open a database session and run the migration script I’m working on, and when it completes, it will leave me inside the open transaction where I can inspect the results of my work. I make changes to the script, roll back the transaction, and run the script again, for as many iterations as it takes. I only add a COMMIT when I’ve tested the whole file and think it’s ready for the next step in testing.

I mentioned above that the customer may want to use this opportunity to clean their data. You should want this, too. Make sure the data you’re feeding your new system is as clean and well-structured as possible. You may find, as we do, that most of the work in your migrations is in validating the input data, and that actually creating new records in your application is almost an afterthought. That’s OK. You may also find there are places where your application, wonderful though it may be, could stand to be more strict about the data it accepts. I have often discovered my application’s database needs a uniqueness constraint, or a foreign key, thanks to a migration I was working on.

Data Migration History

I wish I could truthfully claim all our migrations go off flawlessly, but that would be a lie. It’s not unheard of to run into some corner case, a few weeks or even months after the migration goes live, where data wasn’t migrated correctly.

On the other hand, it’s certainly not uncommon for a customer or coworker to spot something that strikes them as odd, after the migration goes live, only to find later that everything was in fact correct. In either case, it’s important to preserve a history of the migration to investigate these concerns. We accomplish this with a few specific steps:

Create a database schema for the migration, and a table within that schema for each data file or object type we’re importing. I call these tables “staging tables”, where the incoming data is “staged” as it’s cleaned and validated. Having a separate schema means these tables can remain in the production database long after the migration is complete, generally without interfering with anything.
These staging tables should generally use text fields, to be as forgiving and flexible as possible with the incoming data. We can clean, parse, and reformat the data after it’s imported.
Don’t change the data in these staging tables; add to the data instead. In other words, if you need to map a value from the legacy system to a different value for your new system, don’t change the column you imported into the staging table; instead, add a new column to the staging table where you’ll store the re-mapped value. If you need to parse a text field into a date (because you followed the instruction to use text fields!), don’t change the type of an imported column; instead, add a new column of date or timestamp type, to store the parsed value. That way, when three months down the road someone discovers that some of the imported records have weird dates, you have all the information you need to determine whether the fault lies with the imported data or some step of the migration progress. Knowing exactly where the fault crept in leaves you that much more empowered to fix it.
Keep track of your migrated records’ primary keys, in the legacy system and the new system. Imagine you’ve just imported your client’s legacy customer list into a staging table. This data includes the legacy system’s primary key. Add a new column to the table for your new system’s primary key, and populate it. Many of our systems use an integer sequence as a primary key, so we’d add a new integer column to the staging table, and fill it with the next values from the sequence. Following this principle will give you several important abilities:
- You can always connect a record in the legacy system with its corresponding record(s) in the new system. If you’ve imported a customer list in this fashion, then when you’re importing the order data later, and each order points to a customer using a legacy customer primary key, you can easily find the correct customer primary key to use in your system.
- You can easily know if a record in your system comes from the migration, or from normal day-to-day business. You will probably use this every time you try to debug something with your migration.
- If you need to remove all imported records and re-import them, you can identify exactly which records those are. This should be only rarely needed.

Finally, document the decision making process, in comments directly in your code. For instance, if you have a table of mappings from one value to another, chances are good you arrived at the final version of that mapping table only after some discussion with the customer. Chances are also good someone’s going to question it later on. It’s helpful to keep a comment around, something like, # Joe Rogers verified this is the correct mapping in the daily standup meeting, 23 Nov 2022. This is especially common if you eventually decide to ignore a certain class of records. /* Rebecca says ignore all records with type = "ARCHIVED", via group email 9 Jan 2023 */ is a very helpful clue when someone comes around wondering where those records went.

Teamwork

My remaining tips apply to almost any programming project. First, use source control, and commit your code to it often. I can’t count how often I’ve been grateful the Git repository had a backup of my work, or made my work accessible to fill some unexpected need on some other system, nor can I count how many times I’ve been stuck because someone else didn’t commit their code so I couldn’t get at it when I needed it. Let’s not talk about how many times I’ve caused someone else to get stuck in the same way… Of course, don’t commit your customer’s data. But you should commit your code, and commit it often.

Finally, where possible, work with someone else. Two programmers reviewing each other’s code and collaborating on solutions are often far better than two programmers working alone, or one programmer working twice as much.

Speaking of working out solutions together, I’d love your help improving this list. What keys have you found are important for data migration projects? I welcome your comments. And if you’re looking for someone to handle a data migration project for you, give us a call!

Nevada State EpiTrax Launch

2022-12-01T00:00:00+00:00

Photo by Adrien Drj

If COVID-19 has taught us anything, it is that the public health landscape can change quickly, and we need a disease surveillance system that is adaptable to support our ever-evolving climate.

Having access to surveillance data for purposes of contact tracing, following trends, and monitoring evolving disease conditions allows health departments to be agile in response. This is a critical component in providing communities with a robust public health infrastructure.

For all these reasons and many more, the State of Nevada embarked on a journey to migrate away from their surveillance system, NBS, to the open-source EpiTrax system created by the Utah Department of Health. Ultimately, the Nevada decisionmakers made this change because they needed to be on one state-wide system and wanted autonomy to customize the system.

Nevada had been exploring this change for a while but due to unforeseen problems the window of time for implementation was incredibly narrow. In the first four months of End Point’s partnership with Nevada, the team was able to accomplish what was believed to be impossible in so short a time.

The Four Month Sprint

Some of the major milestones on the way to the new EpiTrax system included the following:

Building Out Custom Forms

EpiTrax is composed of core and form fields. Core fields are consistent across all conditions. Form fields allow jurisdictions to customize the data system to collect data specific to the needs of their area or the reporting requirements of their funders.

In Nevada, we framed these forms to collect the data outlined in the Message Mapping Guide set forth by the CDC. Collecting form data is the first step in configuring the system to use the NMI module which allows for seamless reporting.

Configuring the System

EpiTrax is an intuitive system that is highly configurable. For example, EpiTrax will show specific fields by condition or condition type, allowing an investigator working on a COVID record to only see vaccines and treatments for COVID which reduces time and error in data entry.

These parameters need to be defined and set up in the system for a large number of items. End Point was able to provide templates and examples from other states and having a framework for this documentation was vital.

Testing the System

Testing sounds easy: “You just log in and make sure the functionality is there.” However, to adequately test a new environment with multiple jurisdictions and a variety of user groups takes time, attention, and follow-through.

Nevada worked with End Point to bring in users from across the state to test both the functionality of the system and disease-specific variables. Then the test failures were discussed daily and the programmers provided options and support for resolving problems. After modifications were made to the environment the second round of testing was implemented.

Given the tight window of time to launch EpiTrax we were building (testing) the ship as we were sailing it and this was only possible given the commitment of everyone on the team.

Training

You cannot launch a statewide system without training the end users, answering questions, and providing a mechanism for ongoing support. End Point provided countless trainings ranging from general use to administrative functionality of the system, and also provided Nevada with a user manual template that they were able to build on and grow to fit their needs.

Electronic Messaging Staging Area (EMSA)

EMSA is an application that processes lab messages and delivers them into EpiTrax. The delivery of these messages is what creates the record for investigation in EpiTrax. Since this was a new process and system to Nevada, End Point worked with the Nevada team to configure the various vocabularies and rules.

Going Live

On September 12 EpiTrax officially launched in Nevada. End Point staff were there both in person and with a consistent virtual presence to work through user concerns, login issues, and workflow questions.

Conclusion

The customization and scope of EpiTrax are crucial components in allowing Nevada to optimize work processes, streamline the multiple data systems being used into one cohesive system, and greatly reduce redundancy.

While switching systems can seem like a daunting and time-consuming task, so is continuing to operate in crisis mode with a system that stops your team from being dynamic. Any change for the better involving hundreds of people and a lot of data is going to take time, normally much longer than 4 months.

The partnership between Nevada and End Point during this fast-paced launch has better positioned the state to use their data to support well-coordinated public health responses to emerging diseases or outbreaks.

CSTE Conference EpiTrax retrospective

2022-06-29T00:00:00+00:00

Last week we were in Louisville, Kentucky for the CSTE Conference. End Point staffed a conference booth to represent the EpiTrax public health surveillance system to a wonderful group of public health experts.

You can read some background about the conference and CSTE in our earlier blog post announcing our plans.

We really enjoyed meeting new friends in person after two years of canceled events due to the pandemic. We spoke with staff from health departments and disease surveillance teams from several state and local jurisdictions, as well as with experts from the CDC and other software and service vendors.

One of the highlights was going around to meet other people staffing booths at the conference. It charged us up to see and hear about all of the interesting and innovative things going on in the public health space at a time when there is so much that needs to be done. We were particularly struck by the efforts being made in onboarding and distributing ELRs and eCRs, areas where the Electronic Message Staging Area (EMSA, which we deploy and support) can complement and enrich those activities.

The open-source disease surveillance and reporting software EMSA and EpiTrax both enjoyed well-deserved attention as we demonstrated them to numerous groups seeking better solutions to serve people in their jurisdictions. People were very interested in the functionality EpiTrax has for case management, encounters, and NMI reporting.

Of course there is a lot more we could say about EpiTrax and EMSA. So, if you didn’t find us at the conference or if you are interested in what EpiTrax and EMSA can do, contact us here online. We are happy to give you a demo!

End Point booth at CSTE 2022 conference

2022-06-09T00:00:00+00:00

Photo by David Barajas

Converging on Kentucky

We are excited to announce that End Point will be attending the CSTE (Council of State and Territorial Epidemiologists) conference in less than two weeks!

We will be running a booth during the 3-day conference in Louisville, Kentucky from Monday, June 20 through Wednesday, June 22, 2022. Our crew is scheduled to include Steve Yoman, Linda King, Ben Goldstein, and me.

Please come visit us in booth #35 to learn more about the EpiTrax Disease Surveillance System and EMSA (Electronic Message Staging Area). Now is the time for public health entities to turn to modern systems to perform disease surveillance, automate the ingestion of electronic laboratory records (ELRs) and electronic case reports (eCRs), report to the CDC through NMI reporting methods, track outbreaks, perform contact management, and do it all with a highly configurable system.

Our booth will feature a VisionPort Mini to showcase EpiTrax and EMSA! VisionPort is our product combining hardware and software to give organizations the ability to create dynamic, shared immersive multimedia experiences. We are delighted when we have opportunities like this to use VisionPort to inform conference attendees about our own offerings such as EpiTrax and EMSA.

We will be happy to demonstrate EpiTrax to you there. Here is a preview screenshot of the application’s disease outbreaks list view:

CSTE

What is CSTE? The Council of State and Territorial Epidemiologists is an organization that works to advance public health policy and epidemiologic capacity. They also provide information, education, and developmental support of practicing epidemiologists in a wide range of areas as well as expertise for program and surveillance efforts.

Here are some of the conference sessions we plan to attend:

Monday, 2:00–3:15 PM: In-Person Concurrent Breakout Sessions: Surveillance / Informatics I: Not Your Grandma’s Old-Fashioned: Data Modernization
Monday, 5:45–6:30 PM: Virtual Concurrent Roundtable Sessions / Networking Surveillance / Informatics II: Electronic Case Reporting and Public Health Agencies—Needs and Directions
Tuesday, 8:45–10:00 AM: In-Person Concurrent Breakout Sessions: Surveillance / Informatics I: Taking off the Blinders: Innovative Methods for Detection and Surveillance
Tuesday, 5:45–6:30 PM: In-Person Concurrent Roundtable Sessions / Networking: Surveillance / Informatics I: CSTE National ELR Workgroup Meet and Greet
Wednesday, 7:30–8:15 AM: In-Person Concurrent Roundtable Sessions / Networking: Surveillance Informatics I: National Notifiable Disease Surveillance System (NNDSS): A Comprehensive Review

Come see what End Point and EpiTrax can do for you! We would love to meet with you in person, a privilege we don’t take for granted these days.

Or if you can’t make the conference, contact us here online.

Data Reporting with EpiTrax

2022-02-16T00:00:00+00:00

Early COVID-19 form-building in Kansas

In March of 2020, many states struggled with disease surveillance systems that were not ready for the COVID-19 pandemic. States were forced to wait lengthy periods for their vendors or systems administrators to stand up customized forms for COVID-19 investigations.

State of Kansas informatics staff were able to configure their form within a matter of hours. When the first COVID-19 records came rolling in, the state was ready to go. The Kansas Department of Health and Environment (KDHE) was using EpiTrax, a National Electronic Disease Surveillance System (NEDSS) compliant application created by the state of Utah.

EpiTrax contains a powerful form-building utility that was easily customizable, allowing jurisdictional users to create forms for investigation and reporting purposes. Users could configure questions and value sets based on configurable lists, cutting down on free-text entries and ensuring data quality. Forms could be attached automatically or manually and assigned to specified agencies and conditions.

Screenshot of the form builder

December 2021 updates

Flash forward to December 2021. State of Utah developers rolled out an update to the EpiTrax form builder utility. All the previous version features, such as quick configurability and use of lists, remain, with additions to make investigators’ and informaticians’ lives easier. While the old form module was configurable, it did have room for improvement.

The latest updates include question banks and value set banks that can be used across disease forms. This option allows for consistency across all forms and in reporting and analyzing data. State jurisdictions use system forms and core fields to pull data for CDC reporting purposes. The CDC NMI message mapping guides (MMGs) are the new generation of data transfer. These MMGs often require repeating fields, one option now available in the EpiTrax form builder.

Users can configure forms based on the guide formats. Forms can be shared across conditions, aiding in the use of forms across comorbidities. The new form builder allows users to relate fields to existing core fields and sections. This allows the system to be more flexible and aids in creating a better flow in the user interface. The question formats accommodate many data types including address, phone and dates.

Screenshot of the field type selector

Utah’s experience

End Point staff discussed the new form builder with the Utah Department of Health’s (UDOH) Health Informatics Program Manager, Rachelle Boulton. Rachelle related that UDOH was able to add form fields and configure those fields in the EpiTrax NMI Module within one day when the CDC recently released version 1.1 of their COVID-19 MMG. They reached out to the CDC on December 13th to announce their readiness and were the first state to onboard on December 29th.

She attributed the quick turnaround to the new form builder’s ease of use and to their use of the NMI Module. Rachelle said, “The form builder gives us the opportunity to collect data for CDC reporting requirements and allows us to make changes quickly and send data out quickly.”

When asked what she likes most about the new form builder, Rachelle stated that she appreciates the new question and answer banks and their ability to quickly and efficiently create new forms that are appealing on the user interface.

Screenshot of the question manager

Summary

End Point is collaborating with our current clients to migrate data from the old form builder to the new. We are excited to help them use the new functionality.

Along with this effort, we hope to build a form repository for future clients using the new form enhancements which will allow users to quickly onboard for CDC MMG reporting. These new forms would accommodate any needed state workflow changes or additions and would allow states to implement EpiTrax in a timelier manner.

When asked if she had anything else to add, Rachelle said, “I really like that we try to make EpiTrax not just a surveillance system, but a change management system. We embed these changes in the workflow to make users’ lives easier.”

We at End Point agree with Rachelle. We look forward to the future of EpiTrax and what we can do to help it continue to improve moving forward.

How to use regular expression group quantifiers in PostgreSQL

2022-01-27T00:00:00+00:00

I recently encountered a situation where it was necessary to extract address content from text in HL7 V2 format from a PostgreSQL table’s column. The following example is representative:

||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|

In order to manipulate our example, the address section needs to be extracted from the HL7 V2 message PID segment for patient demographic information. The segments have delimiters for fields (|), components (^), subcomponents (&) and repetition (~).

Our example has only fields and components delimited by pipe (|) and caret (^). The address contains nine components delimited by ^.

I hoped to do this by applying a regular expression (regex) because the address is in a standard format that regex can match with alphanumeric and caret repetition.

Here is my journey figuring out how to match the data I wanted.

Regex pattern in grep

As a test, I got this regex working with the grep command, which successfully extracts the address section from the content:

$ content='||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|'
$ echo "$content" | grep -Eo "([A-Za-z0-9 #'.,/-]*\^){8}[A-Za-z0-9 ]*"
123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York

PostgreSQL regex attempt

The PostgreSQL regexp_matches function supports extraction by pattern-matching data from the content. But when I used the same regex pattern with the regexp_matches function, instead of all eight groups, only the eighth value was returned:

=# SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*', 'g');
 regexp_matches 
----------------
 {^}
(1 row)

More generally, our query is returning the Nth matching group instead of returning all matching groups until the Nth regex group. So if we try to fetch the text matching with 3 groups, the quantifier will return the third field of all match sections instead of the third group itself.

=# SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){3}[A-Za-z0-9 ]*', 'g');
   regexp_matches   
--------------------
 {^}
 {^}
 {"New York City^"}
 {USA^}
(4 rows)

PL/Perl function

Since that doesn’t satisfy our requirements, I tried using a Perl regex through my own PL/Perl function, and got the expected answer:

=# CREATE OR REPLACE FUNCTION perl_regexp_matches (IN str text, IN pattern text) RETURNS text AS $$
    my ($input, $pattern) = @_;
    $output = [$input =~ m/($pattern)/];
    return $output->[0]
$$ LANGUAGE plperl;

=# SELECT perl_regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '([A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                    perl_regexp_matches
------------------------------------------------------------
 123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York
(1 row)

But I researched further for a simple solution to achieve the result without using a custom function and the PL/Perl extension.

PostgreSQL regex solution

In Postgres regex syntax, parentheses () create a numbered capture group which leads to returning the contained matching results.

To get the entire matching data, the regex should have a question mark and a colon (?:) added at the beginning of the regex pattern to create a non-capturing group. Then because no group is capturing, instead the complete match is returned:

=> SELECT regexp_matches('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|', '(?:[A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                         regexp_matches                         
----------------------------------------------------------------
 {"123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York"}
(1 row)

That turns out to be what was happening with my PL/Perl function where m/($pattern)/ captures the entire match, and what grep was doing because of its option -o or --only-matching, which prints the matching part of the lines rather than its default of printing the entire line.

And we can also use the Postgres function substring to return the bare text itself instead of arrays as regexp_matches does:

=# SELECT substring('||121212^^^2^ID 1|676767||SELVA^KUMAR^^^^|19480203|M||B||123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York||123456-7890|||M|NON|4000|' FROM '(?:[A-Za-z0-9 #''.,/-]*\^){8}[A-Za-z0-9 ]*');
                         substring
------------------------------------------------------------
 123456 SAMPLE ROAD^^New York City^NY^12345^USA^H^^New York
(1 row)

Reference

EMSA: Electronic Messaging Staging Area

2020-10-30T00:00:00+00:00

Photo by Lee Roberts, CC BY-SA 2.0

End Point has been involved with developing disease surveillance software for over a decade. One component of that work is EMSA, short for “Electronic Messaging Staging Area”.

The EMSA system we’re currently using was developed by a consortium of states led by Utah, Kansas, and Nevada. It is a PHP web application and can be hosted in the cloud or on-premises as you would any other PHP application.

EMSA:

Is an ingest system for HL7 or CSV-formatted lab reports from doctor offices, labs, and other official reporting groups.
Allows exporting XML output to your disease surveillance system.
Is a web console for managing messages.
Allows you to see messages that are malformed.
Has configurable rules for disease types and message routing; some messages may go directly into a surveillance system while others may be stored for later.
Has configurable “automated workflows” to validate message information and perform tasks.
Can be connected to a BI (Business Intelligence) tool for reporting, or the database can be queried directly.
Works with integration packages like Rhapsody to bring in data from other systems.

End Point has also made custom modifications to EMSA allowing address validation for incoming messages.

EMSA can be deployed along with EpiTrax, the open source tracking system, or as a standalone application connected to a different disease surveillance system.

Benefits of EMSA

Aside from all the features mentioned above, EMSA as an open-source solution brings other benefits:

Reduced cost: Other than development and consulting to help set it up, there are no ongoing licensing costs.
Designed to work as part of the open-source EpiTrax disease surveillance system.
Participating in the open source community that is growing in the healthcare industry.

Help is available!

Interested in talking to us more about EMSA for your disease tracking? Reach out today.

References

Emily Roberts, Rachelle Boulton, Josh Ridderhoff, Theron Jeppson. “Automated Processing of Electronic Data for Disease Surveillance.” Online Journal of Public Health Informatics, 10(1):e2, May 22, 2018. https://doi.org/10.5210/ojphi.v10i1.8903

COVID-19 Support for the Kansas Department of Health and Environment

2020-09-14T00:00:00+00:00

Kansas’s existing EpiTrax system

End Point has worked on Kansas’s disease surveillance systems since 2011. In 2018 we migrated them from their legacy TriSano application to the open source EpiTrax surveillance system created by Utah’s Department of Health. The new EpiTrax system had been in full production for about eight months when COVID-19 cases started to grow in the United States.

COVID-19: Help needed

In March 2020, the Director of Surveillance Systems at the Kansas Department of Health and Environment (KDHE) asked us at End Point to create a web-based portal where labs, hospitals, and ad-hoc testing locations could enter COVID-19 test data. While systems existed for gathering data from labs and hospitals, they needed a way to quickly gather data from the many new and atypical sites collecting COVID-19 test information.

Our approach

Since the portal was intended for people who were unfamiliar with the existing EpiTrax application, we were able to create a new design that was simple and direct, unconstrained by other applications. It required a self-registration function so users could access the system quickly and without administrative overhead, and users needed to understand how to use it without extensive training.

We at End Point agreed to create the portal and dashboard for test results reporters immediately, while planning for later development of additional administrative functions. We used Ruby on Rails and Vue.js to build the portal due to their usefulness for rapid development. The Vue.js front-end JavaScript framework allowed us to quickly put together the portal UI and integrate it with the Rails back-end web services.

Once approved, our team got to work setting up the environment, developing the portal application, and rigorously testing it.

Here are some screenshots of the application:

Portal home page

User registration page

Dashboard

Reporters’ entry screens

The portal was launched on April 30.

Contact tracers support

After the EpiTrax portal for COVID-19 was up and running, Kansas made urgent plans to hire 400 contact tracers. We updated the EpiTrax portal to accommodate this new type of user, showing contact tracers only the data they are authorized to view, while providing them the ability to make limited contact record updates and to create tasks. The project is ongoing, and eventually will be used for all diseases.

EpiTrax for Missouri

Late this Spring, after learning of Kansas’s use of the EpiTrax platform and the powerful capabilities that it provides to the Kansas public health system, the State of Missouri reached out to us at End Point to request their own EpiTrax implementation. End Point and Missouri quickly came to agreement, and since July we have been working together build out a full production EpiTrax surveillance system for them.

Contact us!

To discuss an EpiTrax project, contact us. Our expert team is ready to help you meet your requirements.

Consolidating Multiple SFTP Accounts Into One Master Account

2020-03-16T00:00:00+00:00

Photo by Dan Meyers

Recently, a client implemented a data-intensive workflow to generate various reports and insights from a list of facilities as part of an EpiTrax installation. Because a significant portion of these files contain sensitive healthcare data, they needed to strictly comply with HIPAA. Optimally, facilities should be able to transfer files securely and exclusively to our server. One of the best methods of achieving this is to create individual SSH File Transfer Protocol (SFTP) accounts for each source.

SFTP account

Private SFTP accounts were established for each facility and the data was received at a designated path. At these individual points of contact, a third-party application picks up the data and processes further into the pipeline. The following demonstrates how SFTP accounts are developed and configured:

Create a user group for SFTP accounts:

$ addgroup sftpusers

Configure the following settings in sshd_config (this enables an SFTP account and sets the default location as the home path):

$ vi /etc/ssh/sshd_config
...
# override default of no subsystems
Subsystem       sftp    internal-sftp...

Match Group sftpusers
    ChrootDirectory /home/%u
    AllowTCPForwarding no
    X11Forwarding no
    ForceCommand internal-sftp

Restart SSH server to apply changes:

$ systemctl restart ssh

Create an SFTP user account for a facility and place in a folder on the home path to receive data:

# set new user name
sftpuser=the-new-username
useradd $sftpuser
usermod -g sftpusers -s /usr/sbin/nologin $sftpuser
mkdir -p /home/$sftpuser/INPUT_PATH/
chown -R root:root /home/$sftpuser

Mount multiple accounts to one account

The goal here is to point the data from many facilities to one location, but using a single account and path for multiple sites’ data could result in a breach of security and/or privacy. Mounting the receiving path of a facility’s data onto a single master account and then to a “mount point” with a unique facility name takes care of this issue. The process next consolidates files from individual paths on a master account in one place where the application picks up messages for further processing.

The SFTP accounts and the master account should be attached to the same group. This will permit individual accounts to write on the master account-mounted path. In turn, the master account can read files from the same location. This location now has administrative rights for both the SFTP user the group. Group permission of the mounted folder is set to sftpgroup and user permission is set to the facility account.

# create master user account
$ adduser master
$ passwd master

# Add master user to sftpgroup group
usermod -a -G sftpgroup master
getent group sftpgroup

# MOUNT_PATH and sub folders
mkdir -p /home/master/MOUNT_PATH/{Input,Pickup,Backup,Archive}
chown -R master:master /home/master/MOUNT_PATH

We wrote a script to automate the process of creating an SFTP account, mounting it at the master account path, and adding fstab entries to save the mount in the case of a reboot. The script not only saves time, but also avoids human error when creating accounts for all facilities.

#!/bin/bash

# ./create_sftp_account.sh sftpfacilityone FACILITY_ONE

sftpuser=$1
facility_name=$2

# Create SFTP account and add to sftpgroup
useradd $sftpuser
usermod -g sftpusers -s /usr/sbin/nologin $sftpuser
usermod -a -G sftpgroup $sftpuser
mkdir -p /home/$sftpuser/INPUT_PATH
chown root:root /home/$sftpuser

# Create path specific to facility in master account Input folder
mkdir -p /home/master/MOUNT_PATH/Input/$facility_name
chown -R $sftpuser:sftpgroup /home/master/MOUNT_PATH/Input/$facility_name

# Mount sftp account into master account and set permissions
mount --bind /home/master/MOUNT_PATH/Input/$facility_name /home/$sftpuser/INPUT_PATH
chown -R $sftpuser:sftpgroup /home/$sftpuser/INPUT_PATH
chmod g=rwxs /home/$sftpuser/INPUT_PATH

# Add fstab entry to persist and mount on reboot
echo "/home/master/MOUNT_PATH/Input/$facility_name /home/$sftpuser/INPUT_PATH none bind 0 0" >> /etc/fstab
echo "Created user $sftpuser at $facility_name mount point successfully"

Files at one location

Now, data files from facilities are available at individual folders under MOUNT_PATH/Input on the master account. This enables third-party applications to pick up files in a straightforward way to proceed with further processing. It also helps our client access the files for review from the master account easily without navigating into each separate account.

Summary

Mounting multiple SFTP accounts onto one master account turns out to be an efficient and beneficial method of consolidating data. Both safe and secure, running separate SFTP accounts establishes an exclusive private link between facilities and servers. The master account has the unique ability to access files belonging to each facility in order to process the data further.

Tip: In order to avoid broken mounts, check the status by using the command mount -fav. Problems with mount configurations can cause broken mounts after rebooting the server.

Cooking with CAS

2020-03-10T00:00:00+00:00

Photo by Flickr user reidrac, licensed under CC BY-SA 2.0

One of our customers asked us to host a new suite of web-based applications for them and to protect them with a single sign-on (SSO) solution. Ok, easy enough; these applications were in fact designed with a particular SSO system in mind already, but our situation required a different one, and we eventually chose Apereo’s open source Central Authentication Server project, or CAS. I’d like to describe the conversion process we went through.

The ingredients

Our customer’s application suite included:

The principal Java application using JAAS authentication
Another Java application based on Spring Security
A pair of PHP applications
A few automated tasks that needed to authenticate.

The original SSO system sets a header on each request, identifying an authenticated user. This requires a gateway system to sanitize request headers to ensure malicious users cannot forge a header themselves. It also requires each application inspect request headers and respond appropriately.

CAS is a bit more complex: applications redirect unauthenticated requests to a CAS server, which authenticates the user through any of various configurable methods. The CAS server then redirects the user back to the original application with a parameter called a “Service Ticket”, a seemingly random number identifying an individual authentication request. The original application contacts the CAS server directly to validate the service ticket and to collect information to identify the user. It can then establish a session for that user, and proceed normally.

To CAS-enable an application, we incorporate one of the CAS client libraries, which exist for various languages. In fact we won’t use the Java client directly, but rather we’ll incorporate components that extend it. When evaluating CAS, I was a bit concerned by what appeared to be a surprisingly limited selection of actively supported client libraries, and of course your results may vary, but we found software to meet our own needs without too much difficulty.

Configuring Wildfly Authentication

The most important application in the suite depends on the JAAS-based security subsystem of the Wildfly application server it’s deployed to. Originally it used a custom LoginModule that inspected request headers to find the ID of the authenticated user. Our first task was to configure our proxy server to remove this header from every request. We planned to disable the old authentication system entirely, of course, but this change ensured that even if we mistakenly left it enabled somewhere, malicious users couldn’t exploit it for access.

This application uses a declarative security policy: the deployment descriptor identifies a set of user roles, and another set of “security constraints”. Each security constraint describes one or more URL patterns used in the application, and the set of user roles allowed to access URLs matching those patterns. Here are two examples:

<!-- clients -->
<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Area</web-resource-name>
        <url-pattern>/views/clients/*</url-pattern>
        <url-pattern>/client/edit/*</url-pattern>
        <url-pattern>/client/edit_tree/*</url-pattern>
        <url-pattern>/client/view/*</url-pattern>
        <url-pattern>/client/view/id/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>client_role</role-name>
    </auth-constraint>
</security-constraint>

<!-- places -->
<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Area</web-resource-name>
        <url-pattern>/views/places/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>manage_places</role-name>
    </auth-constraint>
</security-constraint>

We installed the cas-extension library in our Wildfly server to handle the CAS protocol. When an unauthenticated user attempts to access a URL matching a pattern in one of the application’s security constraints, the CAS extension automatically intercepts control and redirects the user to the CAS server. Assuming the user authenticates successfully, our application will receive another request with a service ticket parameter. The cas-extension intercepts this request as well, validates the service ticket, and creates an “identity assertion”, which it sends to the Wildfly security system. Wildfly’s role mapper queries a database to find the user’s roles, after which the authentication process is complete.

Configuration of the cas-extension begins with a CAS profile, a combination of the URL of the CAS server and the URL of the service the extension should protect.

<subsystem xmlns="urn:soulwing.org:cas:1.0">
    <cas-profile name="default" service-url="https://our.application.server/application" server-url="https://our.cas.server"/>
</subsystem>

This tells the extension where to send authentication requests, where to listen for requests returning from the CAS server, and where to validate service tickets. Next we need to validate that identity assertion, and figure out what roles belong to the user. This happens in a Wildfly security domain.

<security-domain name="MySecurityDomain">
    <authentication>
        <login-module name="IdentityAssertion" code="org.soulwing.cas.jaas.IdentityAssertionLoginModule" flag="required" module="org.soulwing.cas"/>
    </authentication>
    <mapping>
        <mapping-module code="DatabaseRoles" type="role">
            <module-option name="dsJndiName" value="java:/comp/env/jdbc/databaseConnection"/>
            <module-option name="rolesQuery" value="select role from user_roles where user_id = ?"/>
        </mapping-module>
    </mapping>
</security-domain>

The authentication portion of the security domain refers to a JAAS LoginModule shipped with the CAS extension, which simply verifies that the identity assertion comes from the CAS extension and not somewhere else. Then the mapping portion (documented here) looks up the given user in a database to find what roles it should be assigned.

The last piece of the puzzle is a CAS deployment descriptor for our application, which activates cas-extension for that application. In its simplest form, this is an empty file in the right place, but ours ended up a little more complex. It identifies both the CAS profile we want to use (unnecessary in this case, as there’s only one CAS profile on the system, but it helped keep things more clear in our minds), and instructs the extension to load some other libraries into our application.

<?xml version="1.0" encoding="UTF-8"?>
<cas xmlns="urn:soulwing.org:cas:1.0">
  <profile>default</profile>
  <add-api-dependencies/>
</cas>

This configuration proved sufficient to let users log in and use the application, but as is sometimes the case with single sign-on, we needed a little more work to let them log out properly. Each application in the suite sets a cookie in the user’s browser to identify its session. The CAS server likewise sets a cookie. When a user logs out of the application, that application’s session cookie is destroyed, but we also need to destroy the CAS server’s session cookie as well as the other applications’ cookies. Single log-out can be complicated, and I won’t go into the full setup here. One reason the CAS deployment descriptor loads cas-extension libraries was so we could use cas-extension to generate the proper logout URL, and redirect our users to that URL once the application has destroyed its own session.

Configuring Spring Authentication

Another Java-based application in our suite uses Spring Security. This document describes the bulk of the configuration, which seemed less straightforward than for cas-extension, but follows essentially the same mechanism. Here we configured a Spring UserDetailsService to execute the same database query we used above to find the user’s roles. I’m not fully conversant in the large stack of beans Spring uses to manage the process, and it took some time to get this configuration sorted out.

Enter PHP

Two of these applications use PHP, which meant yet another configuration. Apereo maintains a PHP client, which includes several helpful examples. I tracked down the part of the application that authenticates users, and replaced the existing code with calls to phpCAS:

phpCAS::client(CAS_VERSION_2_0, $phpCASHost, $phpCASPort, $phpCASContext);
phpCAS::forceAuthentication();
$_SESSION[EXPORT_SERVERNAME]['umdid'] = phpCAS::getUser();

The forceAuthentication call determines the current phase of the authentication process this request is in, whether it’s unauthenticated, fully authenticated, or requires service ticket validation, and responds appropriately. We then set a session variable to the ID of the authenticated user, which replicates what the original authentication code would have done.

One of these two applications requires authenticated access to a REST API exposed by the Wildfly application. CAS calls this “proxy” authentication, when one application requests access to another. Here, CAS issues not only its usual service ticket, but also a “proxy granting ticket”. When the application wants to use the API, it asks the CAS server for a service ticket, using the proxy granting ticket. The CAS server itself requires some new configuration in this case, but for the PHP code, the only difference in the login phase from the simpler, non-proxy case is that we call phpCAS::proxy instead of phpCAS::client to configure CAS. Later, when calling the service itself, we used more phpCAS services in place of the cURL library the original used.

$service = \phpCAS::getProxiedService(PHPCAS_PROXIED_SERVICE_HTTP_GET);
$service->setUrl($serviceURL);
$service->send();

The return of header authentication

Finally, we have a few automated tasks which use various APIs, and need to authenticate. We can’t redirect requests to a CAS server and expect a user to provide credentials, so we’ve taken our cue from the applications’ original form, and configured CAS to recognize a “trusted header”. We add this header to any requests issued by these automated jobs. Of course, we’ve also configured the proxy to disallow this header from any systems outside our internal network.

A few loose ends

Of course, there were other considerations in this project that I’ve not covered here. Configuring CAS itself wasn’t necessarily straightforward, and included a custom authentication module I hope to describe in a later article. Selecting among CAS server’s available deployment options and fitting the winner into our existing infrastructure in a way that makes it easy to manage and monitor was another task entirely. We needed to customize the server’s default user interface to prevent it from offering users an imaginary method to recover forgotten passwords. The series of redirected browser requests involved in the CAS protocol presents a notable performance impact under some circumstances. And it has taken me no small effort to learn to appreciate the CAS server’s sometimes distressingly circular documentation. But several months into the project, CAS seems to be working well enough for our purposes that other users of the same application suite have begun to express interest.

How to Migrate from Microsoft SQL Server to PostgreSQL

2019-01-23T00:00:00+00:00

One of our clients had a Java-based application stack on Linux that connected to a pretty old version of SQL Server on Windows. We wanted to migrate the entire system to a more consistent unified stack that developers are efficient with, and that is current so it receives regular updates.

We decided to migrate the database from SQL Server to PostgreSQL on Linux because porting the database, while not entirely quick or simple, was still much simpler than porting the app to .NET/C# would have been. Rewriting the application would have taken far longer, been much riskier to the business, and cost a lot more.

I experimented with a few approaches to the migration and decided to go with the process of schema migration and then the data migration approach which is referred to on the Postgres wiki. Let’s walk through the process of migration step by step.

Schema Migration

A schema of the SQL Server database tables and views needs to be exported to perform schema conversion. The following steps will show you how to export the schema.

Export SQL Server Database Schema

In SQL Management Studio, right click on the database and select Tasks → Generate Scripts.

Choose “Select specific database objects” and check only your application schema Tables (untick dbo schema objects and others if any).

Ensure that “Types of data to script” in advanced options is set to “Schema only”.

Review and save the database tables schema file tables.sql. Use WinSCP and public key auth to transfer tables.sql to the Linux server.

Convert Schema from SQL Server to Postgres

sqlserver2pgsql is a good migration tool written in Perl to convert SQL Server schemas to Postgres schemas. Clone it from GitHub to your database server and execute the following commands to convert the tables schema:

$ git clone https://github.com/dalibo/sqlserver2pgsql.git
$ cd sqlserver2pgsql
$ perl sqlserver2pgsql.pl -f tables.sql -b tables-before.sql -a tables-after.sql -u tables-unsure.sql

The converted schema will be available in tables-before.sql and statements to create constraints will be in tables-after.sql to execute after data migration. Just review tables-unsure.sql and do what’s needed if there are any tables not converted by the tool. If you want to change any schema names in Postgres, you can rename them now in the SQL file, e.g.:

$ sed -i ‘s/sql_server_schema/public/g’ *.sql

Setup Postgres Database:

Hopefully, you have your Postgres database ready in your server. If not, install the latest version of Postgres on your server. Then create a Postgres user and database with permissions granted to the user.

CREATE USER <user_name> WITH
  LOGIN
  NOSUPERUSER
  INHERIT
  NOCREATEDB
  NOCREATEROLE
  NOREPLICATION;

CREATE DATABASE <database_name>
    WITH
    OWNER = <user_name>
    ENCODING = 'UTF8'
    LC_COLLATE = 'en_US.UTF-8'
    LC_CTYPE = 'en_US.UTF-8'
    TABLESPACE = pg_default
    CONNECTION LIMIT = -1;

Once the database and user account are ready, load the converted tables-before.sql script into your database to create tables. Then we can move forward with data migration.

psql -U user_name -p 5432 -h localhost -d database_name -f tables-before.sql
#	or
database_name=# \i /path/to/tables-before.sql

Data Migration

Data migration through data dump and restore makes the process cumbersome, with a huge manual process, data type mismatches, date formats, etc. It is wise to use existing stable systems instead of investing more time into it.

Pentaho Data Integration

Pentaho offers various stable data-centric products. Pentaho Data Integration (PDI) is an ETL tool which provides great support for migrating data between different databases without manual intervention. The community edition of PDI is good enough to perform our task here. It needs to establish a connection to both the source and destination databases. Then it will do the rest of work on migrating data from SQL server to Postgres database by executing a PDI job.

Download Pentaho Data Integration Community Edition and extract the tarball in your local environment. See:

Run spoon.sh to open the GUI application in your local environment.

$ cd pentaho/data-integration
$ ./spoon.sh

Create connections to both SQL Server and Postgres databases in PDI.

1. Create a New Job.

File → New → Job

2. Create Source Database Connection.

Click View in left sidebar → Right Click ‘Database Connections’ → Choose New → Provide SQL Server connection details

3. Create Destination Database Connection.

Click View in left sidebar → Right Click ‘Database Connections’ → Choose New → Provide Postgres connection details

4. From Wizard menu, choose Copy Tables Wizard.

Tools → Wizard → Copy Tables

5. Choose Source and Destination databases.

6. Select the list of tables to migrate.

7. Move forward to choose the path for job and transformation files.

8. The transformations were created to copy data from source to destination database.

9. Copy the Pentaho Data Integration tarball and job with transformations to the Postgres server to avoid network latency on data migration.

$ tar -cvzf pdi-migration-job.tar.gz pdi-migration-job/
$ rsync -azP data-integration.zip pdi-migration-job.tar.gz user@server:

10. Check Postgres server to SQL Server database access (based on architecture design at your end).

Establish SSH tunneling to connect to SQL Server through the application server.

$ cat config
host tunnel-server
    Hostname <application_server>
    Port 22
    User root
    LocalForward 1433 <ipaddress>:1433

$ ssh tunnel-server
$ telnet localhost 1433

PDI Job Execution

Execute PDI job using Pentaho kitchen utility on the database server. Add a kettle configuration to avoid Pentaho considering empty values as NULL values which affects NOT NULL constraints.

$ cat .kettle/kettle.properties
KETTLE_EMPTY_STRING_DIFFERS_FROM_NULL=Y

Confirm IP address for database servers’ connections and execute the job using kitchen.sh, a command-line utility to execute PDI jobs.

$ ./kitchen.sh -file="/path/to/pdi-migration-job.kjb" -level=Basic | tee pdi-migration-job.log

The data migration process took around 30 minutes to copy 10GB of data from SQL Server to Postgres database over network.

Output:

2018/08/23 11:16:52 - SQLServerToPostgres20180823 - Job execution finished
2018/08/23 11:16:52 - Kitchen - Finished!
2018/08/23 11:16:52 - Kitchen - Start=2018/08/23 10:51:00.701, Stop=2018/08/23 11:16:52.951
2018/08/23 11:16:52 - Kitchen - Processing ended after 25 minutes and 52 seconds (1552 seconds total).

We ran into a space problem while copying a large table and handled it by reducing the number of rows size in the corresponding table transformation file. Here’s a discussion of the problem.

Fix:

vi **table_name**.ktr
    <size_rowset>100</size_rowset>

sed 's/<size_rowset>10000<\/size_rowset>/<size_rowset>100<\/size_rowset>/' *table_name*

Once the data migration is completed, execute the tables-after.sql schema-migration script to apply constraints.

Migrate Views

Follow the same steps to convert the views.sql schema. Right click on the database and click on Tasks → Generate Script → Export only views.

$ perl sqlserver2pgsql.pl -f views.sql -b views-before.sql -a views-after.sql -u views-unsure.sql

Migrate Functions

The functions migration requires skill and syntax awareness of both SQL Server and Postgres. The functions need to validated and tested properly after rewriting for Postgres. Place all rewritten functions into postgres-function.sql and load into the database as part of the migration process.

Most Common Errors

ERROR: syntax error at or near "["
Remove SQL Server square brackets quoting around column and table names.
SELECT [user_id] → SELECT user_id
ERROR: operator does not exist: character varying = integer
Type casting on comparison
AND column_name in (1, -1) → AND column_name in (‘1’, ‘-1’)
ERROR: syntax error at or near ","
Datatype conversion
convert(varchar(10),column_name) → column_name::text
ERROR: column "user_id" of relation "user" does not exist
No double quotes on mixed-case column names (unless used everywhere)
INSERT INTO user (“user_id”) VALUES (1) → INSERT INTO user (user_id) VALUES (1)
ERROR: column "mm" does not exist
DATEADD → INTERVAL
DATEADD(mm,-6,GETDATE()) → CURRENT_DATE - INTERVAL ‘6 months’
ERROR: column "varchar" does not exist
Convert date to USA format
convert(varchar, tc_agreed_dt, 101) → to_char(CURRENT_TIMESTAMP, ‘MM/DD/YYYY’)

Result

After migration, we performed extensive testing on the application and optimized the queries in various modules. We also performed a benchmark test on speed and performance between the new and old systems. The migration paid off with a promising result of 40% average gain on overall performance of the application. That reflects the newer server hardware as well as the query optimization work, not solely the move to Postgres.

The entire migration was automated with a script to test multiple times and reduce manual intervention to perform error-free process. The automated script is available to customize based on your scenario: sqlserver-postgres-migration-automation.sh

TriSano Case Study

2017-01-24T00:00:00+00:00

Overview

End Point has been working with state and local health agencies since 2008. We host disease outbreak surveillance and management systems and have expertise providing clients with the sophisticated case management tools they need to deliver in-house analysis, visualization, and reporting—combined with the flexibility to comply with changing state and federal requirements. End Point provides the hosting infrastructure, database, reporting systems, and customizations that agencies need in order to service to their populations.

Our work with health agencies is a great example of End Point’s ability to use our experience in open source technology, Ruby on Rails, manage and back up large secure datasets, and integrate reporting systems to build and support a full-stack application. We will discuss one such client in this case study.

Why End Point?

End Point is a good fit for this project because of our expertise in several areas including reporting and our hosting capabilities. End Point has had a long history of consultant experts in PostgreSQL and Ruby on Rails, which are the core software behind this application.

Also, End Point specializes in customizing open-source software, which can save not-for-profit and state agencies valuable budget dollars they can invest in other social programs.

Due to the secure nature of the medical data in these database, we and our clients must adhere to all HIPAA and CDC policies regarding hosting of data handling, server hosting, and staff authorization and access auditing.

Team

Steve Yoman

Steve serves as the project manager for both communication and internal development for End Point’s relationship with the client. Steve brings many years in project management to the table for this job and does a great job keeping track of every last detail, quote, and contract item.

Selvakumar Arumugam

Selva is one of those rare engineers who is gifted with both development and DevOps expertise. He is the main developer on daily tasks related to the disease tracking system. He also does a great job navigating a complex hosting environment and has helped the client make strides towards their future goals.

Josh Tolley

Josh is one of End Point’s most knowledgeable database and reporting experts. Josh’s knowledge of PostgreSQL is extremely helpful to make sure that the data is secure and stable. He built and maintains a standalone reporting application based on Pentaho.

Application

The disease tracking system consists of several applications including a web application, reporting application, two messaging areas, and SOAP services that relay data between internal and external systems.

TriSano: The disease tracking web app is an open source Ruby on Rails application based on the TriSano product, originally built at the Collaborative Software Initiative. This is a role-based web application where large amounts of epidemiological data can be entered manually or by data transfer.

Pentaho: Pentaho is a PostgreSQL reporting application that allows you to run a separate reporting service or embed reports into your website. Pentaho has a community version and an enterprise version, which is what is used on this particular project. This reporting application provides OLAP services, dashboarding, and generates ad hoc and static reports. Josh Tolley customized Pentaho so that the client can download or create custom reports depending on their needs.

Two Messaging Area applications: The TriSano system also serves as the central repository for messaging feeds used to collect data from local health care providers, laboratories throughout the state, and the CDC.

SOAP services run between the TriSano web app, the Pentaho reporting application, and the client’s data systems translate messages into the correct formats and relay the information to each application.

Into the Future

Based on the success over 9+ years working on this project, the client continues to work with their End Point team to manage their few non open-source software licenses, create long term security strategies, and plan and implement all of their needs related to the continuous improvement and changes in epidemiology tracking. We partner with the client so they can focus their efforts on reading results and planning for the future health of their citizens. This ongoing partnership is something End Point is very proud to be a part of and we hope to continue our work in this field well into the future.

Analyzer Reports with Geo Map Option in Pentaho 5 BI Server

2014-09-12T00:00:00+00:00

The “Geo Map” option in Analyzer Reports provides a feature to visualize data with geographic locations. We will learn how to design a Mondrian schema and configure Pentaho to make use of the “Geo Map” feature in the Analyzer Reports. This article will show us how to set this feature up step by step.

Enable Geo Map feature on Geographic fields in Mondrian Schema

The Mondrian schema has two main categories called Dimensions and Measures. The Dimensions are defined as levels in the Mondrian schema. The Geographic fields should have two additional annotations to use Geo Map. The two annotations are:

Data.Role — defines the type of level generally; for this type of node, this must be set to ‘Geography’.
Geo.Role — defines the geographical classification in a hierarchy. These can be either predefined roles (‘country’, ‘state’, ‘city’, ‘postalcode’) or custom roles.

Sample Level with Annotation:

<Level name="Country Name" visible="true" column="country" type="String" uniqueMembers="false" levelType="Regular" hideMemberIf="Never">
  <Annotations>
    <Annotation name="Data.Role"><![CDATA[Geography]]></Annotation>
    <Annotation name="Geo.Role"><![CDATA[country]]></Annotation>
  </Annotations>
</Level>

Geographic fields and datasets in database

I have created a sample table with the fields containing geographic locations for dimensions and aggregated value for measures. The sample population table contains Pentaho-defined geographic locations ‘country’, ‘state’, ‘city’ and aggregated population count for those geographic fields.

‘Population’ table design and datasets:

Here we create a sample population table with geographic fields and the population count in a PostgreSQL database.

CREATE TABLE population (
   id INT PRIMARY KEY   NOT NULL,
   country      TEXT    NOT NULL,
   state        TEXT    NOT NULL,
   city         TEXT   NOT NULL,
   count        INT    NOT NULL
);

Next we load population data into the table for 4 cities of 2 states in USA. (Population data for more geographic locations in USA are available at USA Population.)

# SELECT * FROM population;
 id | country |   state    |     city      |  count 
----+---------+------------+---------------+---------
  1 | USA     | California | Los Angeles   | 3857800
  2 | USA     | California | San Francisco |  825863
  3 | USA     | New York   | Hilton        |    5974
  4 | USA     | New York   | Johnsburg     |    2390

Download the sql dump file with table schema and datasets.

Design a Mondrian Schema with Geographic Support

Pentaho provides a tool called “Schema Work Bench” to design a Mondrian schema for a specific table’s data. We can create a new Mondrian schema for the table by selecting File -> New -> Schema. The picture below depicts the hierarchy level of the Mondrian schema elements.

Publish the Mondrian schema to Pentaho

The publish process requires the JDBC datasource to have access to database. Create a JDBC datasource in the manage datasources wizard with necessary input values.

Once the JDBC datasource has been created in Pentaho server, the Mondrian schema can be published from the Schema Work Bench.

Download the Mondrian schema xml to view the schema, cube, table, dimension, hierarchy, level, annotations, measures elements and corresponding attribute values.

The Mondrian schema xml can be imported directly into Pentaho server to create an analysis datasource.

Create a Analyzer Report with Geo Map

Add the necessary geographic fields under “Rows” and population count under “Measure” to create a basic analyzer report.

Change the report type to “Geo Map” through the right top corner options to view the visualized data. Congratulations, you’re done!

Install Pentaho BI Server 4.8 Community Edition with PostgreSQL Repository

2013-11-07T00:00:00+00:00

Pentaho BI server community edition can be installed through an archive file available from SourceForge.

Prerequisites

Java 6
PostgreSQL

Download Pentaho BI Server installation file (biserver-ce-4.8.0-stable.zip) from SourceForge: http://sourceforge.net/projects/pentaho/files/Business%20Intelligence%20Server/4.8.0-stable

Unzip the archive file and navigate inside biserver-ce to set sh files to executable mode:

$ unzip biserver-ce-4.8.0-stable.zip
$ cd biserver-ce
$ find . -type f -iname '*.sh' -exec chmod a+x {} \;

Pentaho community edition uses hsql database as default. Need to create two databases in Postgres for Pentaho. Find the SQL files to create databases under biserver-ce/data/postgresql. database_name, user_name and password are configurable through SQL files. Fix two errors before creating database using SQL files. Comment two lines in below files tables as commented.

create_quartz_postgresql.sql

ALTER TABLE qrtz_fired_triggers
    ALTER TRIGGER_NAME  TYPE VARCHAR(200),
    ALTER TRIGGER_GROUP TYPE VARCHAR(200),
    ALTER INSTANCE_NAME TYPE VARCHAR(200),
    ALTER JOB_NAME      TYPE VARCHAR(200),
    ALTER JOB_GROUP     TYPE VARCHAR(200),
    ADD COLUMN PRIORITY INTEGER NULL;
--    ADD COLUMN PRIORITY INTEGER NOT NULL;

migrate_quartz_postgresql.sql

ALTER TABLE qrtz_fired_triggers
    ALTER TRIGGER_NAME  TYPE VARCHAR(200),
    ALTER TRIGGER_GROUP TYPE VARCHAR(200),
    ALTER INSTANCE_NAME TYPE VARCHAR(200),
    ALTER JOB_NAME      TYPE VARCHAR(200),
    ALTER JOB_GROUP     TYPE VARCHAR(200);
--    ADD COLUMN PRIORITY INTEGER NOT NULL;
--    ALTER COLUMN PRIORITY SET NULL;

Modify database name, username & password if necessary and create databases for configuration and scheduling by running below commands.

$ psql -U postgres -a -f create_quartz_postgresql.sql
$ psql -U postgres -a -f create_repository_postgresql.sql
$ psql -U postgres -a -f create_sample_datasource_postgresql.sql
$ psql -U postgres -a -f migrate_quartz_postgresql.sql
$ psql -U postgres -a -f migration.sql

Verify databases quartz (scheduling) and hibernate (configuration) and their tables.

Now database name, username, password and driver should be configured in following places in files. By default hsql drivers, settings enabled in config files so comment hsql configurations and enable Postgres settings.

biserver-ce/tomcat/webapps/pentaho/META-INF/context.xml

<?xml version="1.0" encoding="UTF-8"?>
<Context path="/pentaho" docbase="webapps/pentaho/">
        <Resource name="jdbc/Hibernate" auth="Container" type="javax.sql.DataSource"
                factory="org.apache.commons.dbcp.BasicDataSourceFactory" maxActive="20" maxIdle="5"
                maxWait="10000" username="hibuser" password="password"
                driverClassName="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/hibernate"
                validationQuery="select 1" />
               
        <Resource name="jdbc/Quartz" auth="Container" type="javax.sql.DataSource"
                factory="org.apache.commons.dbcp.BasicDataSourceFactory" maxActive="20" maxIdle="5"
                maxWait="10000" username="pentaho_user" password="password"
                driverClassName="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/quartz"
                validationQuery="select 1"/>
</Context>

biserver-ce/pentaho-solutions/system/applicationContext-spring-security-jdbc.xml

        <bean id="dataSource"
                class="org.springframework.jdbc.datasource.DriverManagerDataSource">
                <property name="driverClassName" value="org.postgresql.Driver" />
                <property name="url" value="jdbc:postgresql://localhost:5432/hibernate" />
                <property name="username" value="hibuser" />
                <property name="password" value="password" />
        </bean>

biserver-ce/pentaho-solutions/system/applicationContext-spring-security-hibernate.properties

jdbc.driver=org.postgresql.Driver
jdbc.url=jdbc:postgresql://localhost:5432/hibernate
jdbc.username=hibuser
jdbc.password=password

hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect

biserver-ce/pentaho-solutions/system/hibernate/hibernate-settings.xml

<config-file>system/hibernate/postgresql.hibernate.cfg.xml</config-file>

biserver-ce/pentaho-solutions/system/hibernate/postgresql.hibernate.cfg.xml

<property name="connection.driver_class">org.postgresql.Driver</property>
    <property name="connection.url">jdbc:postgresql://localhost:5432/hibernate</property>
    <property name="dialect">org.hibernate.dialect.PostgreSQLDialect</property>
    <property name="connection.username">hibuser</property>
    <property name="connection.password">password</property>

biserver-ce/pentaho-solutions/system/simple-jndi/jdbc.properties

Hibernate/type=javax.sql.DataSource
Hibernate/driver=org.postgresql.Driver
Hibernate/url=jdbc:postgresql://localhost:5432/hibernate
Hibernate/user=hibuser
Hibernate/password=password
Quartz/type=javax.sql.DataSource
Quartz/driver=org.postgresql.Driver
Quartz/url=jdbc:postgresql://localhost:5432/quartz
Quartz/user=pentaho_user
Quartz/password=password

Specify the pentaho solutions path, server url and port in web.xml of tomcat webapp

biserver-ce/tomcat/webapps/pentaho/WEB-INF/web.xml

        <context-param>
                <param-name>solution-path</param-name>
                <param-value>/opt/avr-new/biserver-ce/pentaho-solutions</param-value>
        </context-param>

        <context-param>
                <param-name>fully-qualified-server-url</param-name>
                <param-value>http://localhost:8080/pentaho/</param-value>
        </context-param>

Pentaho can be run tomcat custom ports by modifying the ports in server.xml and web.xml

biserver-ce/tomcat/conf/server.xml

    <Connector URIEncoding="UTF-8" port="9090" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443" />

biserver-ce/tomcat/webapps/pentaho/WEB-INF/web.xml

         <context-param>
                <param-name>fully-qualified-server-url</param-name>
                <param-value>http://localhost:8080/pentaho/</param-value>
        </context-param>

Let’s start the Pentaho BI server and try out its great features. Commands to start and stop the BI server:

biserver-ce$ ./start-pentaho.sh
biserver-ce$ ./stop-pentaho.sh

Troubleshooting:

biserver-ce$ tail -f tomcat/logs/catalina.out
biserver-ce$ tail -f tomcat/logs/pentaho.out

Install Pentaho BI Server 5 Enterprise Edition with PostgreSQL repository

2013-11-07T00:00:00+00:00

Pentaho provides different ways to install Pentaho BI server. Each method has its own flexibility in installation.

Installer mode—Easy to install BA & DI server & tools in one flow with default PostgreSQL repo & default Tomcat server. (New Postgres installed on port 5432.)
Archive mode—BA server installed with own BA repository & default Tomcat server. Necessary tools need to be installed manually.
Manual mode—BA server installed with own BA repository & own application server (Tomcat or JBoss). Necessary tools need to installed manually.

We have a Postgres instance running on our server and are good with Tomcat as application server so Archive mode of installation is suitable for us. Pentaho installation requires two things be installed before starting with Pentaho installation.

Java 7
PostgreSQL

Archive mode installation files can be accessible only to license purchased users. Download biserver-ee-5.x-dist.zip from Pentaho customer portal with account credentials here: https://support.pentaho.com/forums/20413716-Downloads

Unzip the archive file and you can see the installation files inside extracted directory.

$ unzip biserver-ee-5.x-dist.zip
$ cd biserver-ee-5.x;ls
install.bat  installer.jar  install.sh  license.txt  README.txt

In remote servers Pentaho can be installed on console mode with -console’. Accept the license and provide the installation path to install Pentaho BI server.

$ java -jar installer.jar -console

Find biserver-ee directory under the installation path and set sh files to executable mode.

$ cd biserver-ee;
$ find . -type f -iname '*.sh' -exec chmod a+x {} \;

Let’s create repository databases by running queries in SQL files located at biserver-ee/data/postgresql.

quartz, hibernate and jackrabbit databases will be created by executing these SQL files. Database names, usernames and password can be changed by modifying in SQL files if required.

$ cd biserver-ee/data/postgresql
$ psql -U postgres -p 5432 -a -f create_jcr_postgresql.sql
$ psql -U postgres -p 5432 -a -f create_quartz_postgresql.sql
$ psql -U postgres -p 5432 -a -f create_repository_postgresql.sql
$ psql -U postgres -p 5432 -a -f pentaho_mart_postgresql.sql

Pentaho uses postgresql as default database and files are configured to use postgresql. So just verify the database_name, username and password to work with installed postgresql and databases created.

biserver-ee/pentaho-solutions/system/quartz/quartz.properties

org.quartz.jobStore.driverDelegateClass = org.quartz.impl.jdbcjobstore.PostgreSQLDelegate

biserver-ee/pentaho-solutions/system/hibernate/hibernate-settings.xml

    <config-file>system/hibernate/postgresql.hibernate.cfg.xml</config-file>

biserver-ee/pentaho-solutions/system/hibernate/postgresql.hibernate.cfg.xml

    <property name="connection.driver_class">org.postgresql.Driver</property>
    <property name="connection.url">jdbc:postgresql://localhost:5432/hibernate</property>
    <property name="dialect">org.hibernate.dialect.PostgreSQLDialect</property>
    <property name="connection.username">pentaho_user</property>
    <property name="connection.password">password</property>

There are more occurrences in this file. Carefully do the necessary changes in all the places.

biserver-ee/pentaho-solutions/system/jackrabbit/repository.xml
biserver-ee/pentaho-solutions/system/jackrabbit/repository/workspaces/default/workspace.xml
biserver-ee/tomcat/webapps/pentaho/META-INF/context.xml

<Resource name="jdbc/PDI_Operations_Mart" auth="Container" type="javax.sql.DataSource"
            factory="org.apache.commons.dbcp.BasicDataSourceFactory" maxActive="20" maxIdle="5"
            maxWait="10000" username="pentaho_user" password="password"
            driverClassName="org.postgresql.Driver" url="jdbc:postgresql://localhost:5432/hibernate"
            validationQuery="select 1"/>

Download postgresql and h2 drivers then place it under biserver-ee/tomcat/lib

postgresql-9.x.jdbc4.jar

h2-1.2.x.jar

Change Tomcat port on these two files to run Pentaho on different port.

Specify the Pentaho solutions path, server URL and port in web.xml of Tomcat webapp.

biserver-ee/tomcat/webapps/pentaho/WEB-INF/web.xml

        <context-param>
                <param-name>solution-path</param-name>
                <param-value>$INSTALLATION_PATH/biserver-ee/pentaho-solutions</param-value>
        </context-param>

        <context-param>
                <param-name>fully-qualified-server-url</param-name>
                <param-value>http://localhost:8080/pentaho/</param-value>
        </context-param>

Pentaho can be configured to run on different ports by changing ports on Tomcat server.xml and web.xml

biserver-ee/tomcat/biserver-ee/tomcat/server.xml

    <Connector URIEncoding"UTF-8" port"9090" protocol"HTTP/1.1"
               connectionTimeout"20000"
               redirectPort"8443" />

- biserver-ee/tomcat/webapps/pentaho/WEB-INF/web.xml
         <context-param>
                <param-name>fully-qualified-server-url</param-name>
                <param-value>http://localhost:9090/pentaho/</param-value>
        </context-param>

Install license

A license needs to be installed to use Pentaho. Navigate to the license-installer directory in installation path. Feed license file to install_license.sh, separating more than one license file with spaces.

$ ./install_license.sh install ../license/Pentaho\ BI\ Platform\ Enterprise\ Edition.lic

Install plugins

Archive mode of installation installs only BI Server. Necessary plugins can to be installed manually. Here install the plugins for reporting, analyzer and dashboard. Plugins are available at the same place where download BI server. Download these three files and place at any path on server:

Reporting—pir-plugin-ee-5.x-dist.zip
Analyzer—pdd-plugin-ee-5.0.0.1-dist.zip
Dashboard—paz-plugin-ee-5.0.0.1-dist.zip

All the plugins installed through same procedure.

Unzip the plugins and navigate to extracted directory
run installer on console, accept the license and provide $INSTALLATION_PATH/biserver-ee/pentaho-solutions/system as location to install plugins

$ java -jar installer.jar -console

Let’s start the BI server

biserver-ee$ ./start-pentaho.sh

Install the licenses for the plugins by login as admin user (default—Admin:password) or install through the command line:

Administration -> License -> install licenses for plugin by click +

Troubleshooting:

biserver-ee$ tail -f tomcat/logs/catalina.out
biserver-ee$ tail -f tomcat/logs/pentaho.log

If the pentaho.xml is present at tomcat/conf/Catalina directory, delete it. It will be generated again when you start the BA Server.

Start and stop the BI server:

biserver-ee$ ./start-pentaho.sh
biserver-ee$ ./stop-pentaho.sh

Dimensional Modeling

2013-05-08T00:00:00+00:00

People occasionally bring up the question, “What exactly is a data warehouse?” Though answers to this question vary, in short a data warehouse exists to analyze data and its behavior over large swaths of time or many different data sources. There’s more to it, though, than simply cramming historical data into the same old database. There are a number of defining characteristics, including the following:

Query patterns and behavior
Data retention policy
Database structure

Query Behavior

Data warehouses are sometimes called “OLAP” databases, which stands for “on-line analytical processing”, in contrast to the more common “OLTP”, or “on-line transaction processing” databases that manage data for an online storefront, a bug tracker, or a blog. A typical OLTP database supports applications that issue short, simple queries, and expect quick answers and support for many simultaneous transactions. The average OLAP query, by contrast, is generally read-only, but can be quite complex, and might take minutes or hours to complete. Such queries will often include heavy-duty statistical processing and data mining, involving terabytes of data.

Data Retention

In a typical e-commerce database, eventually it becomes helpful to archive away older data that the front-end applications won’t need anymore. This helps performance, simplifies backups, and has the nice side-effect of leaving less data available for nefarious black hats that come snooping around. But it doesn’t make sense simply to discard much of this data, because it contains valuable information: customer behavior, supplier response time, etc. Often this deleted data remains alive, in a different form perhaps, in a data warehouse, which can contain data spanning many years.

Database structure

Because query patterns in data warehouses differ so much from OLTP databases, it makes sense to structure the OLAP database to support its queries better. OLTP databases typically follow an “entity” model, hence the ubiquitous (if often not particularly useful) entity-relationship diagram. In such a design, tables represent objects stored in the database, such as an order, a user, or a product. OLAP databases, on the other hand, are commonly “dimensionally” modeled, which results in something called a “star schema”. In a star schema, the database contains large “fact” tables, full of foreign keys pointing to a set of “dimension” tables; when diagrammed, this looks like star with the fact table in the center, or for the more mechanically oriented, a wheel with the fact table at the hub and dimension tables at the end of each spoke. Rather than modeling a particular entity, the fact table generally describes business processes, such as customer conversion or shipping efficiency.

Dimensional modeling is an interesting topic full of its own rules of thumb, which often differ quite dramatically from typical entity modeling. For instance, whereas many database modelers complain about the use of surrogate keys in OLTP databases, it’s recommended in dimensional modeling. Dimensional databases generally don’t need OUTER joins, and rarely contain NULL values. As a result, business intelligence applications designed for data warehousing can make certain assumptions about how they’ll be asked to query the database. These assumptions obviously place some limits on the types of queries the database can process effectively. However it is through these assumptions that the system gains most of its efficiency.

Biggest among the limitations of an OLAP database is what’s called the “grain”. The grain describes exactly what information the fact table contains, and at what level of detail; it should be made clear during the first stages of warehouse design and widely understood by all involved. Queries that require information that isn’t part of the grain, or at finer levels of detail, must find a different fact table to use. But for queries which depend only on the available data, the fact table can be very efficient, as the database can partition it easily, and scan it unencumbered by simultaneous writes from other transactions, and filtered by simple conditions and INNER joins to the various dimension tables.

Data warehouses differ from the traditional database in several other ways, but this covers some of the basics. Dimensional modeling alone is a well-developed field of study with numerous intricacies, where experience and careful training are important for developing a useful final model. But the analytical power of such databases has been proven.

TriSano and Pentaho at our NYC company meeting

2012-06-13T00:00:00+00:00

Josh Tolley just spoke to us about the TriSano open source project he works on. It helps track and report on public health events, using data at least partly gathered from doctors following special legal reporting requirements to look for epidemics.

A lot of this is about data warehousing. Public health officials collect a lot of data and want to easily report on it. Typically they use SPSS. Need to filter the data before doing analysis.

And what is a data warehouse? Store all your data in a different way that’s efficient for querying broken down by time (OLAP). Such queries don’t usually work very well in normal transactional (OLTP) database.

Dimension tables: E.g. different public health departments, sex, disease. Fact tables: Contains the numbers, facts that you may do math aggregation against, and links to dimension tables. The key to the whole process is deciding what you want to track.

Pentaho is what we use for the query interface. To get done what we need to do, we have to make use of unpublished APIs, using JRuby to interface between Pentaho (Java) and TriSano, a Rails app. Postgres is the database.

Brian Buchalter then took over and delved more into the TriSano Rails side of the project.

Let’s talk about the challenge of maintaining Rails applications. Getting them to work today is fine. Getting them to be flexible and change over time is a challenge. The early benefit of Rails was organizational: Having empty directories to put things into. But starting to use standard classic object oriented patterns is finally happening more in Rails.

Brian recommended Sandi Metz’s talk about the path to better design, and her upcoming book, Practical Object Oriented Design in Ruby.

Ruby lets you open a class anytime, anywhere, and do whatever you want to it: “monkey patching.” For example, plugins hijacking public methods. It can cause trouble. He strongly recommends watching the video Capability vs. Suitability by Gary Bernhardt, presented at MountainWest Ruby Conf 2012.

Practical limits unveil over time as people customize in ways you don’t expect. For example, a 1300-question form that takes 4 minutes to load. Add arbitrary limits up front so that people understand there’s a design limit rather than a “bug” of bad performance or UI due to unforeseen use cases.

Software is supposed to be easy to change. That’s what people like, and that’s what our employment is all about. But antipatterns emerge that make change harder over time.

How do we work with our clients when there’s technical debt accumulated that will make further development increasingly expensive and troublesome? How do we estimate with imperfect information? Important to keep yourself from losing focus by diving into implementation details.

If you see part of your app that needs to be implemented but has no classes, just helper calls, double your estimate.

If you see lots of conditional blocks in your code, double your estimate. All the additional logic paths greatly increase complexity and make it harder to test.

Don’t Repeat Yourself (DRY) is of course needed. But refactoring code is not (just) about removing duplication: It’s about making code easier to modify forever.

Refactoring needs to be built into the development cycle, or else we end up forever dealing with first-implementation code, simply moving on to the next project.

Comprehensive tests that take 75 minutes to run won’t get run by developers very often! The project automatically runs tests on check-in, but getting the result back takes too long. Would be good to speed up tests.

Lots of good topics to follow up on here.

Spring authentication plugin

2012-02-24T00:00:00+00:00

One of our clients regularly deploys Pentaho with their application, and wanted their users to be able to log in to both applications with the same credentials. We could, of course, have copied the user information from one application to another, but Pentaho, and the Spring system it uses for authentication, allows us to be much more elegant.

Spring is often described as an “application development framework”, or sometimes an “Inversion of Control container”, which essentially means that you can redefine, at run time, exactly which objects perform various services within your application. That you have to navigate a bewildering and tangled web of configuration files in order to achieve this lofty goal, and that those files suffer from all the verbosity you’d expect from the combined forces of XML and Java, normally isn’t as loudly proclaimed. Those inconveniences notwithstanding, Spring can let you do some pretty powerful stuff, like in our case, redefining exactly how a user gets authenticated by implementing a few classes and reciting the proper incantation in the configuration files.

Spring handles most of the plumbing in the Pentaho authentication process, and it all starts with this particular definition from one of the Spring configuration files:

The authenticationManager bean contains a list of authentication processors, and here’s where our custom code begins. These beans must implement the Spring AuthenticationProvider interface, and to customize the authentication we can just put a new bean in this list. The list is ordered; refer to the ProviderManager documentation for details, but essentially any of the beans in the list can accept a set of credentials. Here, we’ve simply added a reference to our new bean, to the beginning of the list. The new bean is indicated by name only; it needs to be defined elsewhere, like this:

Here we’ve mapped the bean name to a specific class name. We can also add whatever configuration properties we need. So long as there are corresponding setter methods in the actual code (e.g. “public void setProperty_1(String value)”) these will Just Work. Along with these setter methods, we must implement the two methods in Spring’s AuthenticationProvider interface. The most important is the aptly-named authenticate(), which Spring will call when a user tries to log in.

authenticate() gets one argument, an Authentication object, and when a user presents valid credentials, that’s also what it returns. The first thing our new implementation needs to do is get the username and password from the authentication object:

At this point we’ve got two jobs to do: first, validate the credentials, and second (assuming the credentials are valid), determine what roles the user should be given. In this particular case we used the authentication database from the client’s application to validate the credentials; that code will be completely different from one bean to the next, so I won’t share it here, but the second part, finding the user’s roles, is pretty consistent. Spring lets you set up as many roles as you’d like, giving each a different text name, and simply assumes all the different roles will mean something to the application (Pentaho, in this case) later on. For our setup, we look up the user’s roles from a database, and then tell Spring about them like this:

At this point, the user has successfully logged in. If we didn’t want to allow the user in, we could instead throw an AuthenticationException, and Spring would go on to next AuthenticationProvider in the list. The beauty of it all is now that I have a working plugin, I can modify my application’s authentication system simply by modifying a configuration file or two (or three) and restarting the application.