API to convert workflow format to prompt format

[jamesWalker55]

When using the API, it uses a different format than workflow JSON files. This format can be retrieved from EXIF data in saved images, however it can be inconvenient to 1) build a workflow, then 2) generate an image, 3) load the EXIF data from the image, finally 4) use the new prompt in the API.

I would like to suggest an API endpoint that receives a workflow JSON object, converts it to a prompt and returns it. This way, the whole process can be easily automated with a Python script by making a request to that endpoint, then using the returned JSON to make API requests.

[pydn]

Click the settings wheel, then "enable dev mode options" in the pop up. This enables a save (API Format) button.

[guinoise]

This functionnality is required for the sake of simplicity in api consumer.

To use correctly the prompt api we have to send:

1. client_id: this is not an issue
2. prompt : the api json
3. extra_data: { extra_pnginfo: { workflow json }

The third point is required to:

1. Embed the workflow in the generated images
2. Some nodes require this information to work

As of today, this mean you have to export the workflow.json and the workflow_api.json, then do the changes in BOTH files and send them to the api.

If a functionality to convert the workflow to workflow_api or even better an option to push the workflow.json to the prompt API exists, it will make the life of the API consumers a lot easier.

[IntenseSunlight]

I used this simple, hacky way to get the full prompt for the api (with "extra_data"). I agree there should be a better way to get this with the normal "Save (api_format)" operation.

In ./server.py, line 450+ at the time of this writing

        .
        .
        @routes.post("/prompt")
        async def post_prompt(request):
            print("got prompt")
            resp_code = 200
            out_string = ""
            json_data =  await request.json()
            json_data = self.trigger_on_prompt(json_data)
            with open("last_prompt.json", "w") as f:              # <- dump prompt to file
                json.dump(json_data, f, indent=4)                 #  in local directory

            if "number" in json_data:
            .
            . (etc.)

[dmarx]

looks like the graphToPrompt() function is probably the conversion code we're all looking for.

https://github.com/comfyanonymous/ComfyUI/blob/master/web/scripts/app.js#L1857-L1989

[imperator-maximus]

we have a large external application and do a lot of ComfyUI graph generation there and we have to use the normal format. But now we need to execute the workflow of course which is impossible by that internal API format without doing that same conversion on our side.
So I suggest a refactoring doing that conversion only on server-side in Python so both formats could be accepted there.

[RyoJerryYu]

looks like the graphToPrompt() function is probably the conversion code we're all looking for.

https://github.com/comfyanonymous/ComfyUI/blob/master/web/scripts/app.js#L1857-L1989

For those coming from search engines (like me): Since the merge of #4379 , ComfyUI's main branch has removed the original frontend code. The frontend code referenced here refers to:

ComfyUI/web/scripts/app.js

Lines 1857 to 1989 in 951a206

	/**
	* Converts the current graph workflow for sending to the API
	* @returns The workflow and node links
	*/
	async graphToPrompt() {
	for (const outerNode of this.graph.computeExecutionOrder(false)) {
	if (outerNode.widgets) {
	for (const widget of outerNode.widgets) {
	// Allow widgets to run callbacks before a prompt has been queued
	// e.g. random seed before every gen
	widget.beforeQueued?.();
	}
	}
	const innerNodes = outerNode.getInnerNodes ? outerNode.getInnerNodes() : [outerNode];
	for (const node of innerNodes) {
	if (node.isVirtualNode) {
	// Don't serialize frontend only nodes but let them make changes
	if (node.applyToGraph) {
	node.applyToGraph();
	}
	}
	}
	}
	const workflow = this.graph.serialize();
	const output = {};
	// Process nodes in order of execution
	for (const outerNode of this.graph.computeExecutionOrder(false)) {
	const skipNode = outerNode.mode === 2 || outerNode.mode === 4;
	const innerNodes = (!skipNode && outerNode.getInnerNodes) ? outerNode.getInnerNodes() : [outerNode];
	for (const node of innerNodes) {
	if (node.isVirtualNode) {
	continue;
	}
	if (node.mode === 2 || node.mode === 4) {
	// Don't serialize muted nodes
	continue;
	}
	const inputs = {};
	const widgets = node.widgets;
	// Store all widget values
	if (widgets) {
	for (const i in widgets) {
	const widget = widgets[i];
	if (!widget.options || widget.options.serialize !== false) {
	inputs[widget.name] = widget.serializeValue ? await widget.serializeValue(node, i) : widget.value;
	}
	}
	}
	// Store all node links
	for (let i in node.inputs) {
	let parent = node.getInputNode(i);
	if (parent) {
	let link = node.getInputLink(i);
	while (parent.mode === 4 || parent.isVirtualNode) {
	let found = false;
	if (parent.isVirtualNode) {
	link = parent.getInputLink(link.origin_slot);
	if (link) {
	parent = parent.getInputNode(link.target_slot);
	if (parent) {
	found = true;
	}
	}
	} else if (link && parent.mode === 4) {
	let all_inputs = [link.origin_slot];
	if (parent.inputs) {
	all_inputs = all_inputs.concat(Object.keys(parent.inputs))
	for (let parent_input in all_inputs) {
	parent_input = all_inputs[parent_input];
	if (parent.inputs[parent_input]?.type === node.inputs[i].type) {
	link = parent.getInputLink(parent_input);
	if (link) {
	parent = parent.getInputNode(parent_input);
	}
	found = true;
	break;
	}
	}
	}
	}
	if (!found) {
	break;
	}
	}
	if (link) {
	if (parent?.updateLink) {
	link = parent.updateLink(link);
	}
	if (link) {
	inputs[node.inputs[i].name] = [String(link.origin_id), parseInt(link.origin_slot)];
	}
	}
	}
	}
	let node_data = {
	inputs,
	class_type: node.comfyClass,
	};
	if (this.ui.settings.getSettingValue("Comfy.DevMode")) {
	// Ignored by the backend.
	node_data["_meta"] = {
	title: node.title,
	}
	}
	output[String(node.id)] = node_data;
	}
	}
	// Remove inputs connected to removed nodes
	for (const o in output) {
	for (const i in output[o].inputs) {
	if (Array.isArray(output[o].inputs[i])
	&& output[o].inputs[i].length === 2
	&& !output[output[o].inputs[i][0]]) {
	delete output[o].inputs[i];
	}
	}
	}
	return { workflow, output };
	}

BTW, I'm still unable to find a program-friendly way to convert from workflow.json to workflow_api.json , except that implement the graphToPrompt() in my code. I'm really looking forward to seeing this feature implemented on the backend of ComfyUI.

[oumad]

looks like the graphToPrompt() function is probably the conversion code we're all looking for.
https://github.com/comfyanonymous/ComfyUI/blob/master/web/scripts/app.js#L1857-L1989

For those coming from search engines (like me): Since the merge of #4379 , ComfyUI's main branch has removed the original frontend code. The frontend code referenced here refers to:

ComfyUI/web/scripts/app.js

Lines 1857 to 1989 in 951a206

	/**
	* Converts the current graph workflow for sending to the API
	* @returns The workflow and node links
	*/
	async graphToPrompt() {
	for (const outerNode of this.graph.computeExecutionOrder(false)) {
	if (outerNode.widgets) {
	for (const widget of outerNode.widgets) {
	// Allow widgets to run callbacks before a prompt has been queued
	// e.g. random seed before every gen
	widget.beforeQueued?.();
	}
	}
	const innerNodes = outerNode.getInnerNodes ? outerNode.getInnerNodes() : [outerNode];
	for (const node of innerNodes) {
	if (node.isVirtualNode) {
	// Don't serialize frontend only nodes but let them make changes
	if (node.applyToGraph) {
	node.applyToGraph();
	}
	}
	}
	}
	const workflow = this.graph.serialize();
	const output = {};
	// Process nodes in order of execution
	for (const outerNode of this.graph.computeExecutionOrder(false)) {
	const skipNode = outerNode.mode === 2 || outerNode.mode === 4;
	const innerNodes = (!skipNode && outerNode.getInnerNodes) ? outerNode.getInnerNodes() : [outerNode];
	for (const node of innerNodes) {
	if (node.isVirtualNode) {
	continue;
	}
	if (node.mode === 2 || node.mode === 4) {
	// Don't serialize muted nodes
	continue;
	}
	const inputs = {};
	const widgets = node.widgets;
	// Store all widget values
	if (widgets) {
	for (const i in widgets) {
	const widget = widgets[i];
	if (!widget.options || widget.options.serialize !== false) {
	inputs[widget.name] = widget.serializeValue ? await widget.serializeValue(node, i) : widget.value;
	}
	}
	}
	// Store all node links
	for (let i in node.inputs) {
	let parent = node.getInputNode(i);
	if (parent) {
	let link = node.getInputLink(i);
	while (parent.mode === 4 || parent.isVirtualNode) {
	let found = false;
	if (parent.isVirtualNode) {
	link = parent.getInputLink(link.origin_slot);
	if (link) {
	parent = parent.getInputNode(link.target_slot);
	if (parent) {
	found = true;
	}
	}
	} else if (link && parent.mode === 4) {
	let all_inputs = [link.origin_slot];
	if (parent.inputs) {
	all_inputs = all_inputs.concat(Object.keys(parent.inputs))
	for (let parent_input in all_inputs) {
	parent_input = all_inputs[parent_input];
	if (parent.inputs[parent_input]?.type === node.inputs[i].type) {
	link = parent.getInputLink(parent_input);
	if (link) {
	parent = parent.getInputNode(parent_input);
	}
	found = true;
	break;
	}
	}
	}
	}
	if (!found) {
	break;
	}
	}
	if (link) {
	if (parent?.updateLink) {
	link = parent.updateLink(link);
	}
	if (link) {
	inputs[node.inputs[i].name] = [String(link.origin_id), parseInt(link.origin_slot)];
	}
	}
	}
	}
	let node_data = {
	inputs,
	class_type: node.comfyClass,
	};
	if (this.ui.settings.getSettingValue("Comfy.DevMode")) {
	// Ignored by the backend.
	node_data["_meta"] = {
	title: node.title,
	}
	}
	output[String(node.id)] = node_data;
	}
	}
	// Remove inputs connected to removed nodes
	for (const o in output) {
	for (const i in output[o].inputs) {
	if (Array.isArray(output[o].inputs[i])
	&& output[o].inputs[i].length === 2
	&& !output[output[o].inputs[i][0]]) {
	delete output[o].inputs[i];
	}
	}
	}
	return { workflow, output };
	}

BTW, I'm still unable to find a program-friendly way to convert from workflow.json to workflow_api.json , except that implement the graphToPrompt() in my code. I'm really looking forward to seeing this feature implemented on the backend of ComfyUI.

@RyoJerryYu Can you share how you implemented that in your code ? my attempts have failed at producing a correct workflow_api.json file, I have imported the litegraph library and copied the graphToPrompt function to use in my app in this manner :

import { LGraph, LGraphEventMode } from '@comfyorg/litegraph';

async function processData(graphData, clean = true) {

  graph = new LGraph()
  graph.start()
  graph.configure(graphData)
  
  // rest of graphToPrompt code without "this."
  // ...
}

I don't know what I'm missing.
By the way the referenced code has moved to the comfyUI front end project here : https://github.com/Comfy-Org/ComfyUI_frontend/blob/b0cc5eac959877fbc72acbb80cbe437c9ada5050/src/scripts/app.ts#L2303

[RyoJerryYu]

@RyoJerryYu Can you share how you implemented that in your code ? my attempts have failed at producing a correct workflow_api.json file, I have imported the litegraph library and copied the graphToPrompt function to use in my app in this manner :

import { LGraph, LGraphEventMode } from '@comfyorg/litegraph';

async function processData(graphData, clean = true) {

  graph = new LGraph()
  graph.start()
  graph.configure(graphData)
  
  // rest of graphToPrompt code without "this."
  // ...
}

I don't know what I'm missing. By the way the referenced code has moved to the comfyUI front end project here : https://github.com/Comfy-Org/ComfyUI_frontend/blob/b0cc5eac959877fbc72acbb80cbe437c9ada5050/src/scripts/app.ts#L2303

Actually, I didn't manage to implement it successfully. It seems to rely heavily on frontend code from custom nodes, and implementing it ourselves would require a significant amount of work. In the end, we chose to modify the main flow of our project, so that it is no longer necessary to convert from workflow.json to workflow_api.json in our backend.