Sweep: Pass server-side error messages through to the API response

[wwzeng1]

Right now any exceptions that occur server-side are not passed through to the API response. Instead, all server-side errors return error code 500 with no additional details. This can be frustrating for errors that are easily resolved, for example expired API keys or invalid values for certain parameters.

We should pass all server-side error messages through to the API response, and use the appropriate error codes depending on the nature of the exception.

[sweep-ai]

Here's the PR! #4.

💎 Sweep Pro: I used GPT-4 to create this ticket. You have 114 GPT-4 tickets left.

• Install Sweep Configs: Pull Request

---

Step 1: 🔍 Code Search

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.

psychic/server/server/main.py

Lines 418 to 598 in 278428b

	min_chunk_size = request.min_chunk_size
	max_chunk_size = request.max_chunk_size
	documents = []
	for connector_id in connector_ids:
	connector = get_document_connector_for_id(connector_id, config)
	if connector is None:
	continue
	result = await connector.load(
	ConnectionFilter(
	connector_id=connector_id,
	account_id=account_id,
	uris=uris,
	section_filter_id=request.section_filter,
	page_cursor=request.page_cursor,
	page_size=request.page_size,
	)
	)
	if chunked and connector_id != ConnectorId.notion:
	raise HTTPException(
	status_code=400, detail="Chunking is only supported for Notion"
	)
	elif chunked:
	chunker = DocumentChunker(
	min_chunk_size=min_chunk_size, max_chunk_size=max_chunk_size
	)
	result = GetDocumentsResponse(
	documents=chunker.chunk(result.documents),
	next_cursor=result.next_page_cursor,
	)
	documents.extend(result.documents)
	response = result
	response.documents = documents
	logger.log_api_call(config, Event.get_documents, request, response, None)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.get_documents, request, None, e)
	raise HTTPException(status_code=500, detail=str(e))
	@app.post(
	"/get-tickets",
	response_model=GetTicketsResponse,
	)
	async def get_tickets(
	request: GetTicketsRequest = Body(...),
	config: AppConfig = Depends(validate_token),
	):
	try:
	account_id = request.account_id
	# If connector_id is not provided, return documents from all connectors
	if not request.connector_id:
	connections = StateStore().get_connections(
	ConnectionFilter(account_id=account_id), config
	)
	if len(connections) == 0:
	raise HTTPException(
	status_code=404, detail="No connections found for this Account"
	)
	connector_ids = [connection.connector_id for connection in connections]
	else:
	connector_ids = [request.connector_id]
	tickets = []
	for connector_id in connector_ids:
	connector = get_ticket_connector_for_id(connector_id, config)
	if connector is None:
	continue
	result = await connector.load_tickets(
	ConnectionFilter(
	connector_id=connector_id,
	account_id=account_id,
	page_cursor=request.page_cursor,
	page_size=request.page_size,
	),
	redact_pii=request.redact_pii,
	)
	tickets.extend(result.tickets)
	response = result
	response.tickets = tickets
	logger.log_api_call(config, Event.get_tickets, request, response, None)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.get_tickets, request, None, e)
	raise HTTPException(status_code=500, detail=str(e))
	@app.post(
	"/get-conversations",
	response_model=GetConversationsResponse,
	)
	async def get_conversations(
	request: GetConversationsRequest = Body(...),
	config: AppConfig = Depends(validate_token),
	):
	# TODO: Add limits to conversations returned
	try:
	connector_id = request.connector_id
	account_id = request.account_id
	oldest_message_timestamp = request.oldest_message_timestamp
	page_cursor = request.page_cursor
	connector = get_conversation_connector_for_id(connector_id, config)
	if connector is None:
	raise HTTPException(status_code=404, detail="Connector not found")
	response = await connector.load_messages(
	account_id,
	oldest_message_timestamp=oldest_message_timestamp,
	page_cursor=page_cursor,
	)
	logger.log_api_call(config, Event.get_conversations, request, response, None)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.get_conversations, request, None, e)
	raise HTTPException(status_code=500, detail=str(e))
	@app.post(
	"/run-sync",
	response_model=RunSyncResponse,
	)
	async def run_sync(
	request: RunSyncRequest = Body(...),
	config: AppConfig = Depends(validate_token),
	):
	try:
	sync_all = request.sync_all
	success = await SyncService(config).run(sync_all=sync_all)
	response = RunSyncResponse(success=success)
	logger.log_api_call(config, Event.run_sync, request, response, None)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.run_sync, request, None, e)
	raise HTTPException(status_code=500, detail=str(e))
	@app.post(
	"/ask-question",
	response_model=AskQuestionResponse,
	)
	async def run_sync(
	request: AskQuestionRequest = Body(...),
	config: AppConfig = Depends(validate_token),
	):
	try:
	# If connector_id is empty, we will use documents from all connectors
	if not request.connector_ids:
	connections = StateStore().get_connections(
	ConnectionFilter(account_id=request.account_id), config
	)
	else:
	connections = []
	for connector_id in request.connector_ids:
	connections.extend(
	StateStore().get_connections(
	ConnectionFilter(
	connector_id=connector_id, account_id=request.account_id
	),
	config,
	)
	)
	result = await QuestionService(config, request.openai_api_key).ask(
	request.question, connections
	)
	response = AskQuestionResponse(answer=result.answer, sources=result.sources)
	logger.log_api_call(config, Event.ask_question, request, response, None)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.ask_question, request, None, e)

psychic/server/server/main.py

Lines 218 to 369 in 278428b

	request: AddSectionFilterRequest = Body(...),
	config: AppConfig = Depends(validate_token),
	):
	try:
	connector_id = request.connector_id
	account_id = request.account_id
	filter = request.section_filter
	for section in filter.sections:
	section.children = None
	connections = StateStore().get_connections(
	ConnectionFilter(connector_id=connector_id, account_id=account_id), config
	)
	if len(connections) == 0:
	raise HTTPException(status_code=404, detail="Connection not found")
	connection = connections[0]
	section_filters = connection.section_filters
	if section_filters is None:
	section_filters = []
	for i in range(len(section_filters)):
	existing_filter = section_filters[i]
	if existing_filter.id == filter.id:
	# remove existing filter
	section_filters.remove(existing_filter)
	section_filters.append(filter)
	StateStore().update_section_filters(
	config,
	connector_id=connector_id,
	account_id=account_id,
	filters=section_filters,
	)
	response = AddSectionFilterResponse(success=True, section_filter=filter)
	logger.log_api_call(config, Event.add_section_filter, request, response, None)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.add_section_filter, request, None, e)
	raise HTTPException(status_code=500, detail=str(e))
	@app.post(
	"/add-section-filter-public",
	response_model=AddSectionFilterResponse,
	)
	async def add_section_filter_public(
	request: AddSectionFilterRequest = Body(...),
	config: AppConfig = Depends(validate_public_key),
	):
	try:
	connector_id = request.connector_id
	account_id = request.account_id
	filter = request.section_filter
	connection = StateStore().load_credentials(
	config, connector_id=connector_id, account_id=account_id
	)
	section_filters = connection.section_filters
	if section_filters is None:
	section_filters = []
	for i in range(len(section_filters)):
	existing_filter = section_filters[i]
	if existing_filter.id == filter.id:
	# remove existing filter
	section_filters.remove(existing_filter)
	section_filters.append(filter)
	StateStore().update_section_filters(
	config,
	connector_id=connector_id,
	account_id=account_id,
	filters=section_filters,
	)
	if connection.new_credential is not None:
	StateStore().add_connection(
	config,
	connection.new_credential,
	connection.connector_id,
	connection.account_id,
	connection.metadata,
	None,
	)
	response = AddSectionFilterResponse(success=True, section_filter=filter)
	logger.log_api_call(config, Event.add_section_filter, request, response, None)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.add_section_filter, request, None, e)
	raise HTTPException(status_code=500, detail=str(e))
	@app.post(
	"/add-apikey-connection",
	response_model=AuthorizationResponse,
	)
	async def add_apikey_connection(
	request: AuthorizeApiKeyRequest = Body(...),
	config: AppConfig = Depends(validate_public_key),
	):
	try:
	connector_id = request.connector_id
	account_id = request.account_id
	credential = request.credential
	metadata = request.metadata
	connector = get_connector_for_id(connector_id, config)
	if connector is None:
	raise Exception("Connector not found")
	result = await connector.authorize_api_key(account_id, credential, metadata)
	response = AuthorizationResponse(result=result)
	logger.log_api_call(
	config, Event.add_apikey_connection, request, response, None
	)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.add_apikey_connection, request, None, e)
	raise HTTPException(status_code=500, detail=str(e))
	@app.post(
	"/add-oauth-connection",
	response_model=AuthorizationResponse,
	)
	async def add_oauth_connection(
	request: AuthorizeOauthRequest = Body(...),
	config: AppConfig = Depends(validate_public_key),
	):
	try:
	auth_code = request.auth_code or None
	connector_id = request.connector_id
	account_id = request.account_id
	metadata = request.metadata
	connector = get_connector_for_id(connector_id, config)
	if connector is None:
	raise Exception("Connector not found")
	result = await connector.authorize(account_id, auth_code, metadata)
	response = AuthorizationResponse(result=result)
	logger.log_api_call(config, Event.add_oauth_connection, request, response, None)
	return response
	except Exception as e:
	print(e)
	logger.log_api_call(config, Event.add_oauth_connection, request, None, e)
	raise HTTPException(status_code=500, detail=str(e))
	@app.post(
	"/update-connection-metadata",
	response_model=UpdateConnectionMetadataResponse,
	)
	async def update_connection_metadata(
	request: UpdateConnectionMetadataRequest = Body(...),
	config: AppConfig = Depends(validate_public_key),

psychic/client/python/psychicapi/psychic.py

Lines 90 to 260 in 278428b

	class GetConversationsResponse:
	messages: List[Dict]
	next_page_cursor: Optional[str] = None
	def __init__(self, messages: List[Dict], next_page_cursor: Optional[str]) -> None:
	self.messages = messages
	self.next_page_cursor = next_page_cursor
	class GetDocumentsResponse:
	documents: List[Dict]
	next_page_cursor: Optional[str] = None
	def __init__(
	self, documents: List[Dict], next_page_cursor: Optional[str] = None
	) -> None:
	self.documents = documents
	self.next_page_cursor = next_page_cursor
	class GetTicketsResponse:
	tickets: List[Dict]
	next_page_cursor: Optional[str] = None
	def __init__(
	self, tickets: List[Dict], next_page_cursor: Optional[str] = None
	) -> None:
	self.tickets = tickets
	self.next_page_cursor = next_page_cursor
	class ChunkingOptions:
	min_chunk_size: Optional[int] = None
	max_chunk_size: Optional[int] = None
	class Psychic:
	def __init__(self, secret_key: str):
	self.api_url = "https://api.psychic.dev/"
	self.secret_key = secret_key
	def handle_http_error(self, response: requests.Response):
	if response.status_code == 401 or response.status_code == 403:
	raise Exception("Unauthorized: Invalid or missing secret key")
	try:
	data = response.json()
	message = data.get("detail", "No additional information")
	except requests.exceptions.JSONDecodeError:
	message = "No additional information"
	raise Exception(f"HTTP error {response.status_code}: {message}")
	def get_documents(
	self,
	*,
	account_id: str,
	connector_id: Optional[ConnectorId] = None,
	section_filter_id: Optional[str] = None,
	uris: Optional[List[str]] = None,
	chunked: Optional[bool] = False,
	min_chunk_size: Optional[int] = None,
	max_chunk_size: Optional[int] = None,
	page_cursor: Optional[str] = None,
	page_size: Optional[int] = 100,
	):
	body = {
	"account_id": account_id,
	"connector_id": connector_id.value if connector_id is not None else None,
	"section_filter_id": section_filter_id,
	"uris": uris,
	"chunked": chunked,
	"min_chunk_size": min_chunk_size,
	"max_chunk_size": max_chunk_size,
	"page_cursor": page_cursor,
	"page_size": page_size,
	}
	response = requests.post(
	self.api_url + "get-documents",
	json=body,
	headers={
	"Authorization": "Bearer " + self.secret_key,
	"Accept": "application/json",
	},
	)
	if response.status_code == 200:
	data = response.json()
	documents = data["documents"]
	next_page_cursor = data["next_page_cursor"]
	return GetDocumentsResponse(
	documents=documents, next_page_cursor=next_page_cursor
	)
	else:
	self.handle_http_error(response)
	def get_connections(
	self,
	*,
	connector_id: Optional[ConnectorId] = None,
	account_id: Optional[str] = None,
	):
	filter = {
	"connector_id": connector_id.value if connector_id is not None else None,
	"account_id": account_id,
	}
	response = requests.post(
	self.api_url + "get-connections",
	json={
	"filter": filter,
	},
	headers={
	"Authorization": "Bearer " + self.secret_key,
	"Accept": "application/json",
	},
	)
	if response.status_code == 200:
	connections = response.json()["connections"]
	for connection in connections:
	connection["connector_id"] = ConnectorId(connection["connector_id"])
	if connection.get("sections"):
	connection["sections"] = [
	Section(**section) for section in connection["sections"]
	]
	if connection.get("section_filters"):
	typed_section_filters = []
	for section_filter in connection["section_filters"]:
	sections = [
	Section(**section) for section in section_filter["sections"]
	]
	id = section_filter["id"]
	typed_section_filters.append(
	SectionFilter(id=id, sections=sections)
	)
	connection["section_filters"] = typed_section_filters
	return [Connection(**connection) for connection in connections]
	else:
	self.handle_http_error(response)
	def add_section_filter(
	self,
	*,
	connector_id: ConnectorId,
	account_id: str,
	section_filter: SectionFilter,
	):
	body = {
	"connector_id": connector_id.value,
	"account_id": account_id,
	"section_filter": {
	"id": section_filter.id,
	"sections": [
	{"id": section.id, "name": section.name, "type": section.type}
	for section in section_filter.sections
	],
	},
	}
	response = requests.post(
	self.api_url + "add-section-filter",
	json=body,
	headers={
	"Authorization": "Bearer " + self.secret_key,
	"Accept": "application/json",
	},
	)
	if response.status_code == 200:
	filter = response.json()["section_filter"]
	filter = SectionFilter(
	id=filter["id"],
	sections=[Section(**section) for section in filter["sections"]],
	)
	return filter

psychic/client/python/psychicapi/psychic.py

Lines 190 to 366 in 278428b

	"connector_id": connector_id.value if connector_id is not None else None,
	"account_id": account_id,
	}
	response = requests.post(
	self.api_url + "get-connections",
	json={
	"filter": filter,
	},
	headers={
	"Authorization": "Bearer " + self.secret_key,
	"Accept": "application/json",
	},
	)
	if response.status_code == 200:
	connections = response.json()["connections"]
	for connection in connections:
	connection["connector_id"] = ConnectorId(connection["connector_id"])
	if connection.get("sections"):
	connection["sections"] = [
	Section(**section) for section in connection["sections"]
	]
	if connection.get("section_filters"):
	typed_section_filters = []
	for section_filter in connection["section_filters"]:
	sections = [
	Section(**section) for section in section_filter["sections"]
	]
	id = section_filter["id"]
	typed_section_filters.append(
	SectionFilter(id=id, sections=sections)
	)
	connection["section_filters"] = typed_section_filters
	return [Connection(**connection) for connection in connections]
	else:
	self.handle_http_error(response)
	def add_section_filter(
	self,
	*,
	connector_id: ConnectorId,
	account_id: str,
	section_filter: SectionFilter,
	):
	body = {
	"connector_id": connector_id.value,
	"account_id": account_id,
	"section_filter": {
	"id": section_filter.id,
	"sections": [
	{"id": section.id, "name": section.name, "type": section.type}
	for section in section_filter.sections
	],
	},
	}
	response = requests.post(
	self.api_url + "add-section-filter",
	json=body,
	headers={
	"Authorization": "Bearer " + self.secret_key,
	"Accept": "application/json",
	},
	)
	if response.status_code == 200:
	filter = response.json()["section_filter"]
	filter = SectionFilter(
	id=filter["id"],
	sections=[Section(**section) for section in filter["sections"]],
	)
	return filter
	else:
	self.handle_http_error(response)
	def get_conversations(
	self,
	*,
	account_id: str,
	connector_id: ConnectorId,
	page_cursor: Optional[str] = None,
	oldest_timestamp: Optional[int] = None,
	):
	body = {
	"connector_id": connector_id.value,
	"account_id": account_id,
	"page_cursor": page_cursor,
	}
	if oldest_timestamp is not None:
	body["oldest_timestamp"] = oldest_timestamp
	response = requests.post(
	self.api_url + "get-conversations",
	json=body,
	headers={
	"Authorization": "Bearer " + self.secret_key,
	"Accept": "application/json",
	},
	)
	if response.status_code == 200:
	data = response.json()
	messages = data["messages"]
	next_page_cursor = data["next_page_cursor"]
	return GetConversationsResponse(
	messages=messages, next_page_cursor=next_page_cursor
	)
	else:
	self.handle_http_error(response)
	def get_conversations(
	self,
	*,
	account_id: str,
	connector_id: ConnectorId,
	page_cursor: Optional[str] = None,
	oldest_timestamp: Optional[int] = None,
	):
	body = {
	"connector_id": connector_id.value,
	"account_id": account_id,
	"page_cursor": page_cursor,
	}
	if oldest_timestamp is not None:
	body["oldest_timestamp"] = oldest_timestamp
	response = requests.post(
	self.api_url + "get-conversations",
	json=body,
	headers={
	"Authorization": "Bearer " + self.secret_key,
	"Accept": "application/json",
	},
	)
	if response.status_code == 200:
	data = response.json()
	messages = data["messages"]
	next_page_cursor = data["next_page_cursor"]
	return GetConversationsResponse(
	messages=messages, next_page_cursor=next_page_cursor
	)
	else:
	self.handle_http_error(response)
	def get_tickets(
	self,
	*,
	account_id: str,
	connector_id: ConnectorId,
	redact_pii: Optional[bool] = False,
	page_cursor: Optional[str] = None,
	):
	body = {
	"connector_id": connector_id.value,
	"account_id": account_id,
	"redact_pii": redact_pii,
	"page_cursor": page_cursor,
	}
	response = requests.post(
	self.api_url + "get-tickets",
	json=body,
	headers={
	"Authorization": "Bearer " + self.secret_key,
	"Accept": "application/json",
	},
	)
	if response.status_code == 200:
	data = response.json()
	tickets = data["tickets"]
	next_page_cursor = data["next_page_cursor"]
	return GetTicketsResponse(
	tickets=tickets, next_page_cursor=next_page_cursor
	)
	else:
	self.handle_http_error(response)

psychic/dashboard/src/pages/query.tsx

Lines 26 to 173 in 278428b

	<h1 className="text-xl font-semibold text-gray-900 dark:text-white sm:text-2xl">
	Query
	</h1>
	</div>
	</div>
	</div>
	<ProductsTable />
	</NavbarSidebarLayout>
	);
	};
	export default QueryPage;
	const ProductsTable: FC = function () {
	const { bearer } = useUserStateContext();
	const [input, setInput] = useState("");
	const [loading, setLoading] = useState(false);
	const [messages, setMessages] = useState<Array<MessageProps>>([
	{
	message: {
	answer: "Hello, how can I help you today?",
	sources: [],
	},
	isUser: false,
	},
	]);
	const messagesEndRef = useRef<HTMLDivElement | null>(null);
	const scrollToBottom = () => {
	messagesEndRef.current?.scrollIntoView({ behavior: "smooth" });
	};
	const getBotResponse = async (message: string) => {
	// send a post request to https://sidekick-server-ezml2kwdva-uc.a.run.app/ with the message
	const response = await fetch(
	"https://sidekick-server-ezml2kwdva-uc.a.run.app/ask-llm",
	{
	method: "POST",
	headers: {
	"Content-Type": "application/json",
	Authorization: `Bearer ${bearer}`,
	},
	body: JSON.stringify({
	queries: [
	{
	query: message,
	top_k: 5,
	},
	],
	possible_intents: [
	{
	name: "question",
	description: "user is asking a question",
	},
	],
	}),
	}
	);
	const data = await response.json();
	console.log(data);
	const answer = data.results[0].answer;
	const sources = data.results[0].results;
	for (var i = 0; i < sources.length; i++) {
	const placeHolder = `Content${i}`;
	// replace the placeholder with the source title
	answer.replace(placeHolder, sources[i].title);
	}
	var dedupedSources = sources.filter((source: any, index: number) => {
	return (
	index ===
	sources.findIndex((obj: any) => {
	return obj.title === source.title && obj.url === source.url;
	})
	);
	});
	var finalResponse = {
	answer,
	sources: dedupedSources,
	};
	setMessages((prevMessages) => [
	...prevMessages,
	{ message: finalResponse, isUser: false },
	]);
	};
	const handleChange = (e: any) => {
	setInput(e.target.value);
	};
	const handleSubmit = async (e: any) => {
	e.preventDefault();
	console.log("Message sent:", input);
	setInput("");
	setMessages((prevMessages) => [
	...prevMessages,
	{ message: { answer: input, sources: [] }, isUser: true },
	]);
	setLoading(true);
	await getBotResponse(input);
	setLoading(false);
	};
	useEffect(() => {
	scrollToBottom();
	}, [messages]);
	return (
	<>
	<div className="h-[500px] w-full bg-gray-200 overflow-y-auto ">
	{messages.map((message, index) => (
	<Message
	key={index}
	message={message.message}
	isUser={message.isUser}
	/>
	))}
	<div ref={messagesEndRef} />
	</div>
	<form onSubmit={handleSubmit} className="p-4 border-t border-gray-300">
	<div className="flex items-center">
	<input
	type="text"
	value={input}
	onChange={handleChange}
	disabled={loading}
	className="flex-grow px-4 py-2 border border-gray-300 rounded-lg focus:outline-none focus:border-blue-500"
	placeholder="Type your message"
	/>
	<button
	type="submit"
	disabled={loading}
	className="ml-4 py-2 px-6 bg-blue-500 text-white rounded-lg focus:outline-none focus:bg-blue-600"
	>
	{loading ? <Spinner size="sm" /> : <div>Send</div>}
	</button>
	</div>
	</form>
	</>
	);
	};

---

Step 2: 🧐 Snippet Analysis

From looking through the relevant snippets, I decided to make the following modifications:

File Path

Proposed Changes

server/server/main.py

In this file, we need to modify the exception handling in all the API endpoint functions. Instead of just printing the exception and returning a generic HTTP 500 error, we should return a more specific error message and status code based on the exception type. This can be achieved by creating a custom exception handler that maps different exception types to appropriate HTTP status codes and error messages. This handler should be used in all server-side functions that interact with the API.

---

Step 3: 📝 Planning

I have created a plan for writing the pull request. I am now working my plan and coding the required changes to address this issue. Here is the planned pull request:

Pass server-side error messages through to the API response
sweep/pass-error-messages

Description

This PR modifies the server-side exception handling in the server/server/main.py file to pass server-side error messages through to the API response. Currently, all server-side errors return a generic HTTP 500 error with no additional details. This can be frustrating for errors that are easily resolved, such as expired API keys or invalid parameter values.

To address this issue, a custom exception handler has been implemented that maps different exception types to appropriate HTTP status codes and error messages. This handler is used in all server-side functions that interact with the API. Now, when an exception occurs, the error message and status code will be included in the API response, providing more informative error messages to the client.

Summary of Changes

• Modified the exception handling in the server/server/main.py file to include specific error messages and status codes based on the exception type.
• Created a custom exception handler that maps different exception types to appropriate HTTP status codes and error messages.
• Used the custom exception handler in all server-side functions that interact with the API.

This change improves the user experience by providing more informative error messages and status codes, making it easier to identify and resolve server-side errors.

---

Step 4: ⌨️ Coding

I have finished coding the issue. I am now reviewing it for completeness.

---

Step 5: 🔁 Code Review

Success! 🚀

---

I'm a bot that handles simple bugs and feature requests but I might make mistakes. Please be kind!
^{Join Our Discord}