Druid Operator is not getting host

[emincanoguz11]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

We use Airflow 2.3.3. I see that this test is successful, but I take a this error. This is the picture

File "/home/airflow/.local/lib/python3.7/site-packages/requests/sessions.py", line 792, in get_adapter
    raise InvalidSchema(f"No connection adapters were found for {url!r}")

I take dag like this

Also I tried this type but I failed

ingestion_2 = SimpleHttpOperator(
    task_id='test_task',
    method='POST',
    http_conn_id=DRUID_CONN_ID,
    endpoint='/druid/indexer/v1/task',
    data=json.dumps(read_file),
    dag=dag,
    do_xcom_push=True,
    headers={
        'Content-Type': 'application/json'
    },
    response_check=lambda response: response.json()['Status'] == 200)

I get this log

[2022-10-13, 06:16:46 UTC] {http.py:143} ERROR - {"error":"Missing type id when trying to resolve subtype of [simple type, class org.apache.druid.indexing.common.task.Task]: missing type id property 'type'\n at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 1]"}

I don't know this is bug or issue or networking problem but can we check this?

P.S - We use Airflow on Kubernetes so that we can not debug it.

What you think should happen instead

No response

How to reproduce

No response

Operating System

Kubernetes

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[emincanoguz11]

P.S I updated Airflow 2.4.0 but it is not work :(

[o-nikolas]

It looks like the druid hook is getting the protocol to use from the conn type, and since you're using a Druid connection that ends up being druid:// in your url, but requests doesn't find any adapters for that protocol:

airflow/airflow/providers/apache/druid/hooks/druid.py

Lines 65 to 69 in 0d78ba5

	host = conn.host
	port = conn.port
	conn_type = conn.conn_type or 'http'
	endpoint = conn.extra_dejson.get('endpoint', '')
	return f"{conn_type}://{host}:{port}/{endpoint}"

It also looks like you're not providing an endpoint, the default seems to be {"endpoint": "druid/indexer/v1/task"}

I have no Druid experience so I can't advise much more than that unfortunately.

[emincanoguz11]

Hi Nikolas,

We saw that this type but I cannot change this. Because we are using Airflow on Kubernetes. But also we gave HTTP Operator to Druid Broker for this reason but it is not worked.

[emincanoguz11]

Hi Nikolas Again,

I tested with standalone Airflow and I tested only http but this is not worked :(

[emincanoguz11]

I solved problem. Thanks but this logs are so terrible to defining this problem. Maybe we will create new logs for defining problem.

[o-nikolas]

Hey @emincanoguz11

I solved problem. Thanks but this logs are so terrible to defining this problem. Maybe we will create new logs for defining problem.

That sounds great! Can you provide more details on what the actual problem was and what logging you think would improve things? That would be massively helpful to others in the future. You could even open a PR yourself and become one of the many contributors to the airflow project! 🚀

[emincanoguz11]

This is ingestion file problem but I see in debug mode for in details. We must see log in Airflow for about ingestion problem. I hope I take PR. Thanks for answering.

[o-nikolas]

I hope I take PR.

Okay, I've assigned this ticket to you for now :)