Update Data Script for US Core v4/v5

[jawalonoski]

Install

This uscore-data-script requires working installations of Ruby and Java. After cloning this repository, run:

cd <path>/uscore-data-script
bundle install
get_validator.sh

Creating the Data Set

1. Run uscore-data-script

The uscore-data-script will execute Synthea to generate synthetic patients, load the resulting
FHIR Bundles, and select certain patients based on criteria described below.

bundle exec ruby uscore-data-script.rb [mrburns] [version]

There is an optional mrburns parameter, which if included, will generate a single longitudinal
patient who possesses at least one resource conforming to every US Core profile. In other words,
one patient with everything. If the mrburns parameter is not included, the script will generate
a small collection of testing patients.

There is an optional version parameter. Legal values are v3, v4, and v5 to specify a version of
the US Core Implementation Guide. The default value is v4.

[Jammjammjamm]

I haven't had any success loading this data into HAPI. Using a v4 mrburns I get:

Error uploading transaction because: {
  "resourceType": "OperationOutcome",
  "issue": [ {
    "severity": "error",
    "code": "processing",
    "diagnostics": "Invalid match URL \"Practitioner?identifier=http://hl7.org/fhir/sid/us-npi|9999542092\" - No resources match this search"
  } ]
}

When I look in output/data I see a 3 transactions, 2 batches, and a Group. Does that seem right? Is output/data still the output I should be using?

[jawalonoski]

I haven't had any success loading this data into HAPI. Using a v4 mrburns I get:
...
When I look in output/data I see a 3 transactions, 2 batches, and a Group. Does that seem right? Is output/data still the output I should be using?

Taking a look at this...

[jawalonoski]

@Jammjammjamm Please re-review this when you are able.

My filtering of data was too aggressive and I was sometimes missing references that were causing errors with HAPI.

One other error that can happen with HAPI is if a conditional reference resolves to more than one resource. See the heading Conditional References under http://hl7.org/fhir/R4/http.html#trules.

Because the HAPI public demo server is... well, public... there are tons and tons of Synthea resources that all sorts of people randomly post there. That means you are likely to have NPI conflicts with Organizations and Practitioners... which means that you should probably test this on your own clean FHIR server or a clean local copy of the hapi-fhir-jpaserver-starter or something equivalent.

Because of reference dependencies, POST the files in output/data in this order:

1. One Bundle of Organizations and Locations
2. One Bundle of Practitioners and PractitionerRoles
3. Bundle(s) of Patient and associated resources
4. One Group

[jawalonoski]

Found a bug in the US Core v5 output... fixing...

[Jammjammjamm]

v4 and v5 are working fine, but I get this when generating a v3 Mr. Burns:

Prefilter constraint testing...
  All constraints satisfied.
  All profiles present.
Filtering selected patient data...

g..................................................................................................eee
  - Filtered 349 resources down to 38 (4 seconds).

d..................................................................................................d.........................................................................................................................................................................a...............
  - Filtered 732 resources down to 79 (4 seconds).
Traceback (most recent call last):
	1: from uscore-data-script.rb:241:in `<main>'
/Users/smacvicar/code/uscore-data-script/lib/filter.rb:127:in `filter!': undefined method `entry' for nil:NilClass (NoMethodError)

[Jammjammjamm]

With a US Core 4 Mr. Burns, I see the following issues:

• Could not find name.suffix, name.use:old, name.period.end in the 3 provided Patient resource(s)
• Could not find statusReason in the 4 provided Immunization resource(s)
• Could not find reportedBoolean, reportedReference, dosageInstruction.text in the 8 provided MedicationRequest resource(s)
• Could not find valueString in the 11 provided Observation resource(s) (Lab Results profile)
• Could not find performedDateTime in the 5 provided Procedure resource(s)
• Could not find these DocumentReference types: 11488-4, 34117-2, 28570-0, 18748-4, 11526-1 (Clinical Notes Guidance)
• Could not find Organization.identifier:NPI in the 1 provided Organization resource(s)
• No resources using the DataAbsentReason Extension have been found

[Jammjammjamm]

I see the following issues with a US Core 5 Mr. Burns:

• Could not find name.suffix, name.use:old, name.period.end, address.use:old, Patient.extension:genderIdentity in the 2 provided Patient resource(s)
• Could not resolve Must Support references participant.member(http://hl7.org/fhir/us/core/StructureDefinition/us-core-relatedperson)
• Could not find result, media, media.link in the 18 provided DiagnosticReport resource(s) (Report and Note Exchange profile)
• Could not find statusReason in the 8 provided Immunization resource(s)
• Could not find reportedBoolean, reportedReference, MedicationRequest.category:us-core in the 15 provided MedicationRequest resource(s)
• Could not find valueCodeableConcept, valueString in the 55 provided Observation resource(s) (Lab Results profile)
• Could not find performer in the 162 provided Observation resource(s) (SDOH Assessment profile).
• Observation.category:survey: minimum required = 1, but only found 0 (from http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-sdoh-assessment) (SDOH Assessment profile, this may be an Inferno issue rather than a data issue)
• Could not find performer in the 159 provided Observation resource(s) (Social History profile)
• Observation.category:us-core/social-history: minimum required = 1, but only found 0 (from http://hl7.org/fhir/us/core/StructureDefinition/us-core-observation-social-history) (Social History profile, this may be an inferno issue rather than a data issue)
• Could not find component, component.code, component.valueQuantity, component.code.coding.code:3151-8, component.valueQuantity.system:http://unitsofmeasure.org, component.valueQuantity.code:L/min, component.code.coding.code:3150-0, component.valueQuantity.value, component.valueQuantity.unit, component.valueQuantity.code:%, Observation.component:FlowRate, Observation.component:Concentration in the 1 provided Observation resource(s) (Pulse Ox profile)
• Could not find valueQuantity, valueCodeableConcept in the 3 provided Observation resource(s) (Imaging Result profile)
• Could not find effectiveDateTime, valueCodeableConcept, valueString in the 2 provided Observation resource(s) (Clinical Test Result profile)
• Could not find performedDateTime in the 79 provided Procedure resource(s)
• Could not find ServiceRequest.category:us-core in the 5 provided ServiceRequest resource(s)
• Could not resolve Must Support references agent.who(http://hl7.org/fhir/us/core/StructureDefinition/us-core-organization)
• No RelatedPerson resources appear to be available.
• No resources using the DataAbsentReason Extension have been found