Data Ingestion
How do I publish data to Hydra?
...
There are a few different ways to verify the publishing of your data:
- the HTTP endpoint (http://{{host}}:{{port}}/streams/{{topic_name}}?start=latest&groupId=test) is probably the easiest way. This endpoint uses chunked transfer encoding, meaning that the connection will be continuously open for as long as a). there is a client connected to it or b). no records are received within a timeout. (
idleTimeoutrequest param, defaults to 60s). - if you're looking for a current “snapshot”, you can download a json file to your local computer by using the
downloadparameter. For example: http://{{host}}:{{port}}/streams/{{topic_name}}?groupId=test&start=earliest&download=true - or you could submit a streaming job to replicate the data, and verify that you're seeing messages you've sent to ingest be replicated (Streaming Data Tutorial).
- the HTTP endpoint (http://{{host}}:{{port}}/streams/{{topic_name}}?start=latest&groupId=test) is probably the easiest way. This endpoint uses chunked transfer encoding, meaning that the connection will be continuously open for as long as a). there is a client connected to it or b). no records are received within a timeout. (
I'm getting errors posting to ingest. Where should I start looking?
- make sure your headers are correct!
- Hydra ingestion relies on several values to validate your data, make sure it gets sent to the right topic, and make sure you have the ability to update and delete messages
- did you evolve your avro schema?
- does your avro schema match your data?
Data Streams
How can I see what data is available in the Platform?
- Go to `{{host}}:{{port}}/schemas`. Find the topic name for the data you want to replicate (e.g., `exp.plans.Plan`). If you go to `/schemas/{topicName}`, the `hydra.key` listed is going to be the primary key in the table that gets created.
How do I replicate or stream data out of Hydra?
How do I verify my streaming job is successful?
- visit the streaming UI, your job should be in the `running` state
- visit the `status` endpoint for your job
How do I reset my streaming job?
- visit the streaming UI get the jobId
- stop the streaming job
- use Postman or another utility to POST to `/jobs/<job-id>/reset`
- start the streaming job from within the streaming UI
my streaming job has stopped replicating/working. Where do I look?
- check the `status` endpoint of your job (ie: https://{{host}}:{{port}}/jobs/{{job_id}}/status)
- check the `logs` endpoint of your job (ie: https://{{host}}:{{port}}/jobs/{{job_id}}/logs)
- did the shape of the data you're replicating change? Hydra has strict rules around schemas and primary keys