[FEAT]: System Logs #528

VipinDevelops · 2024-08-30T02:17:10Z

Acceptance Criteria Fulfillment

Created a System page on the app that checks whether the middleware system is fully healthy. Users should be able to view the related logs directly in the UI without needing to open the terminal or any other tool.

Add all necessary UI components.
Implement server-side event streaming.
Add status updates and stream with a timer.
Implement a watcher for file changes and stream updated content to the client.
Show all logs in real-time.

How it works?

we monitor services (API Server, Redis, Postgres, Sync Server) and stream real-time updates to the client. It checks each service's status through commands or API requests and reads logs when files change. When a service status changes or logs update, the client is notified via a server-sent event stream, keeping the UI updated with the latest information in realtime.

Proposed Changes (including videos or screenshots)

Screencast.from.2024-09-07.23-48-52.webm

samad-yar-khan · 2024-09-06T12:11:27Z

@VipinDevelops why is the grid for log options re-rendering each time I open an overlay and close it ?

Uploading Screen Recording 2024-09-06 at 5.36.19 PM.mov…

samad-yar-khan · 2024-09-06T12:18:55Z

web-server/app/api/stream/route.ts

+      const startWatchers = () => {
+        LOG_FILES.forEach((logFile) => {
+          const watcher = watch(logFile.path, async (eventType) => {
+            if (eventType === 'change' && !streamClosed) {


Put this 'change' string in a const or Enum.

Sure I will Make an Enum

samad-yar-khan · 2024-09-06T12:20:05Z

web-server/src/components/Service/SystemStatus.tsx

+    eventSource.onmessage = (event) => {
+      const data = JSON.parse(event.data);
+
+      if (data.type === 'status-update') {


same, these string literals should be stored as const or Enums

samad-yar-khan · 2024-09-06T12:23:08Z

web-server/src/components/Service/SystemStatus.tsx

+      ) : (
+        <FlexBox col gap={2}>
+          {Object.keys(services).map((serviceName) => {
+            const ServiceName = serviceName as ServiceNames;


var name should be camelcase

VipinDevelops · 2024-09-06T12:39:46Z

@VipinDevelops why is the grid for log options re-rendering each time I open an overlay and close it ?

Uploading Screen Recording 2024-09-06 at 5.36.19 PM.mov…

Interesting I haven't added anything in the overlay that might cause Rerendering no state changes and all, can you share the video you tried uploading, please?

e-for-eshaan · 2024-09-06T13:14:21Z

web-server/src/slices/service.ts

+export type ServiceStatusState = {
+  [key in ServiceNames]: Service;
+};


optional:
can be easily written as Record<ServiceNames, Service>

Sure will do that

e-for-eshaan · 2024-09-06T13:15:51Z

web-server/src/slices/service.ts

+  services: ServiceStatusState;
+  loading: boolean;
+  error?: string;
+  active: string | null;


not string,

saw it being implemented like
const active = useSelector((s) => s.service.active) as ServiceNames;

so initialize it as

active: ServiceNames | null;

then use can remove typecasting from here:
const active = useSelector((s) => s.service.active)

I am removing this state to minimize the rerender I will pass the active service name as prop to the overlay page

e-for-eshaan · 2024-09-06T13:16:51Z

web-server/src/slices/service.ts

+  },
+  active: null,
+  loading: true,
+  error: undefined


undefined is usually never a good state to initliase with.
use null instead.

it means an 'intentional empty value'

Sure will do that

e-for-eshaan · 2024-09-06T13:17:16Z

web-server/src/slices/service.ts

+type SetStatusPayload = {
+  statuses: { [key in ServiceNames]: Status };
+};


optional, use Record<ServiceNames, Status>

e-for-eshaan · 2024-09-06T13:17:55Z

web-server/src/slices/service.ts

+  name: 'services',
+  initialState,
+  reducers: {
+    setActiveService: (state, action: PayloadAction<string | null>) => {


Payload wont't pe string if you implement previous comments.
instead ServiceNames

e-for-eshaan · 2024-09-06T13:26:33Z

web-server/src/components/Service/SystemStatus.tsx

+
+  const { addPage } = useOverlayPage();
+
+  const ServiceTitle: { [key: string]: string } = {


Record<ServiceNames, string>

web-server/src/components/Service/SystemStatus.tsx

e-for-eshaan · 2024-09-06T13:28:00Z

web-server/src/components/Service/SystemStatus.tsx

+                      <Box
+                        component="span"
+                        sx={{
+                          display: 'inline-block',
+                          marginLeft: '6px',
+                          width: '10px',
+                          height: '10px',
+                          borderRadius: '50%',
+                          backgroundColor: isUp ? '#28a745' : '#dc3545'
+                        }}
+                      ></Box>


[optional]:
if empty component, <Box .... />

e-for-eshaan · 2024-09-06T13:28:59Z

web-server/src/constants/stream.ts

+export type ServiceStatus = {
+  [key in ServiceNames]: { isUp: boolean };
+};


optional:
Record<ServiceNames, {isUp: boolean}>

e-for-eshaan · 2024-09-06T13:30:29Z

web-server/src/content/Service/SystemLogs.tsx

+  const containerRef = useRef<HTMLDivElement>(null);
+
+  const services = useSelector(
+    (state: { service: { services: ServiceStatusState } }) =>


why add type to state, it's derived directly, no need to pass it

Will update

e-for-eshaan · 2024-09-06T13:34:31Z

we can implement the CLI color theme for the borders of the log-cards

VipinDevelops · 2024-09-06T13:43:46Z

we can implement the CLI color theme for the borders of the log-cards

For Sure Sounds like a nice idea

e-for-eshaan · 2024-09-06T13:46:31Z

web-server/src/components/Service/SystemStatus.tsx

+                          width: '10px',
+                          height: '10px',
+                          borderRadius: '50%',
+                          backgroundColor: isUp ? '#28a745' : '#dc3545'


i am assuming this is the online-offline dot on the right of log-title

we have some theme colors inbuilt.

search for how we use theme.colors.success.main and theme.colors.error.main in our code-base.

you won't have to create your own colors (mostly!)

Great thanks for letting me know , we can use similar colors for log as well but I was thinking the colors are useful because in CLI we have a lot of logs at once but in this case we only show log for one service at a time does these make sense in here ?

sorry, i don't understand your comment, re-explain?

the colors that you hardcoded (green and red) can be replaced with our own thematic colors of success and error. that's all i wanted you to change

Got you eshan sorry I miss understood

VipinDevelops · 2024-09-06T14:04:18Z

@VipinDevelops why is the grid for log options re-rendering each time I open an overlay and close it ?

Uploading Screen Recording 2024-09-06 at 5.36.19 PM.mov…

Hey @samad-yar-khan You were right Actually I was using a state to update the Active service that caused extra rerenders I removed it and used props instead and now the page render is minimal similar to any other page in the project and it only rerenders when there is a new event pushed and state updated

VipinDevelops · 2024-09-06T16:08:24Z

we can implement the CLI color theme for the borders of the log-cards

Hey @e-for-eshaan this is what you meant right?

Looks good?

e-for-eshaan · 2024-09-06T16:26:22Z

No, i meant borders for these, the green ones can be replaced with their respective colors

VipinDevelops · 2024-09-06T16:40:02Z

Ohk Got it I will update it ASAP

VipinDevelops · 2024-09-06T17:30:10Z

How about now @e-for-eshaan ? you like it Looking colorful right

VipinDevelops · 2024-09-07T02:58:05Z

Hey @e-for-eshaan and @samad-yar-khan ,

Thank you so much for your review—it helped improve the PR's quality alot. I’ve made the requested changes and would appreciate your taking another look.

Thanks again! 😄

VipinDevelops · 2024-09-07T03:42:35Z

I would appreciate your reviews as well, @adnanhashmi09 and @shivam-bit. 😄

shivam-bit

@VipinDevelops please go through Flexbox & Line components with their props.

shivam-bit · 2024-09-07T04:59:32Z

web-server/src/content/Service/SystemLogs.tsx

+
+  useEffect(() => {
+    if (!serviceName) {
+      router.push('/system');


Don't hardcode /system , instead use ROUTES.SYSTEM.PATH.
Check codebase for usage of ROUTES

okay use this ROUTES.SYSTEM_LOGS.PATH

shivam-bit · 2024-09-07T05:02:06Z

web-server/src/content/Service/SystemLogs.tsx

+  const containerRef = useRef<HTMLDivElement>(null);
+  const services = useSelector((state) => state.service.services);
+  const logs = useMemo(() => {
+    return serviceName ? services[serviceName]?.logs || [] : [];


Based on prop types serviceName isn't optional, so it will be always true.
so simply use services[serviceName]?.logs || []

in that case we don't even have to redirect to system-status right

shivam-bit · 2024-09-07T05:02:47Z

web-server/src/content/Service/SystemLogs.tsx

+    <Box
+      ref={containerRef}
+      sx={{
+        padding: '16px 24px',
+        borderRadius: 2,
+        overflowY: 'auto',
+        maxHeight: '750px',
+        whiteSpace: 'pre-wrap',
+        wordBreak: 'break-word',
+        lineHeight: 1.6,
+        marginTop: '8px'
+      }}
+    >


use FlexBox,
Most of the styling will be done automatically.

shivam-bit · 2024-09-07T05:03:20Z

web-server/src/content/Service/SystemLogs.tsx

+            <Typography
+              key={index}
+              style={{
+                marginBottom: '8px',
+                fontFamily: 'monospace',
+                padding: '2px'
+              }}
+            >
+              {log}
+            </Typography>


use our custom <Line> component.
Most of the styling will be done automatically.

shivam-bit · 2024-09-07T05:06:16Z

web-server/src/constants/service.ts

i am quite sure we already have this const map declared somewhere.
in the cli folder for sure, or check resources.ts

@VipinDevelops i think it you can use import { ServiceNames } from '@/constants/service';

@shivam-bit You mean no changes in this we can use it, right?

shivam-bit · 2024-09-07T05:09:09Z

web-server/src/components/Service/SystemStatus.tsx

+                        sx={{
+                          fontWeight: '500',
+                          fontSize: '0.95em',
+                          color: isUp ? '#28a745' : '#dc3545',


use theme.colors.success.main and theme.colors.secondary.main
Import useTheme from mui

shivam-bit · 2024-09-07T05:10:30Z

web-server/src/components/Service/SystemStatus.tsx

+                  <FlexBox col relative fullWidth flexGrow={1}>
+                    <FlexBox
+                      alignCenter
+                      sx={{ width: '100%', paddingTop: '8px' }}


rather then giving width:100% use fullwidth prop on Flexbox
also you can use paddingTop={1} directly on Flexbox

shivam-bit · 2024-09-07T05:13:04Z

web-server/src/components/Service/SystemStatus.tsx

+              <CardRoot
+                key={serviceName}
+                onClick={() => handleCardClick(serviceKey)}
+                sx={{
+                  transition: 'box-shadow 0.2s ease',
+                  '&:hover': {
+                    boxShadow: '0 4px 15px rgba(0, 0, 0, 0.1)'
+                  },
+                  backgroundColor: 'rgba(255, 255, 255, 0.05)',
+                  borderRadius: '12px',
+                  border: `1px solid ${
+                    isUp ? borderColor : alpha(theme.colors.error.main, 0.3)
+                  }`,
+                  padding: '16px',
+                  cursor: 'pointer',
+                  position: 'relative',
+                  overflow: 'hidden'
+                }}
+              >


rather then using CardRoot use Button, it will work same way with most styling issues sorted for you.

shivam-bit · 2024-09-07T05:17:24Z

web-server/pages/system-logs.tsx

+      showEvenIfNoTeamSelected={true}
+      isLoading={loading}
+    >
+      {isGithubIntegrated ? <SystemStatus /> : <Loader />}


i don't think system log has to do anything with github integration.

shivam-bit · 2024-09-07T05:20:52Z

web-server/src/components/Service/SystemStatus.tsx

+        <Box
+          sx={{
+            display: 'flex',
+            alignItems: 'center',
+            justifyContent: 'center'
+          }}
+        >
+          <CircularProgress color="primary" size={50} />
+        </Box>


use <Flexbox alignCenter justifyCenter/> instes of Box`

or <Flexbox centered/>

VipinDevelops · 2024-09-07T05:45:41Z

@VipinDevelops please go through Flexbox & Line components with their props.

Hey @shivam-bit Thanks alot man amazing comments, Here you mean I should Pass props in the FlexBox and Line to provide them styles right ?

shivam-bit · 2024-09-07T06:06:03Z

@VipinDevelops please go through Flexbox & Line components with their props.

Hey @shivam-bit Thanks alot man amazing comments, Here you mean I should Pass props in the FlexBox and Line to provide them styles right ?

yes correct @VipinDevelops

VipinDevelops · 2024-09-07T07:51:43Z

Hey @shivam-bit, thank you so much for suggesting custom components! Using them really takes away the pain of writing a lot of CSS.
I've made the changes you requested—please take a look.
Thanks again! :)

shivam-bit · 2024-09-07T21:19:28Z

web-server/src/components/Service/SystemStatus.tsx

                      <Line
-                        sx={{
-                          fontWeight: '500',
-                          fontSize: '0.95em',
-                          color: isUp ? '#28a745' : '#dc3545',
-                          lineHeight: '1.4'
-                        }}
+                        fontWeight={'500'}
+                        fontSize={'0.95em'}
+                        lineHeight={'1.4'}
+                        white
+                      >
+                        Status:
+                      </Line>
+                      <Line
+                        fontWeight={'500'}
+                        fontSize={'0.95em'}
+                        lineHeight={'1.4'}
+                        color={
+                          isUp
+                            ? theme.colors.success.main
+                            : theme.colors.error.main
+                        }
+                        marginLeft={'8px'}
                      >


@VipinDevelops prefer to use something like <Line bold bigish white>

Ohh yeah forgot this one

shivam-bit · 2024-09-07T21:24:15Z

web-server/src/components/Service/SystemStatus.tsx

+
+  return (
+    <FlexBox col gap={2} padding={'16px'}>
+      <Line bold white fontSize="24px" marginBottom={2}>


rathe then fontSize="24px" use huge

Sure, I thought this one looks too small but if you suggest I will update it

shivam-bit · 2024-09-07T21:26:33Z

web-server/src/components/Service/SystemStatus.tsx

+      <Divider sx={{ mb: 2, backgroundColor: theme.colors.secondary.light }} />
+
+      {loading ? (
+        <FlexBox justifyCenter alignCenter minHeight={'50vh'}>


i don't think anywhere we are showing loader with minHeight={'50vh'} , please look into other places where we are showing a loader, or fill as a flexbox prop

Yep you are correct Sure

VipinDevelops · 2024-09-08T05:03:20Z

Hey @shivam-bit Please take a look, I made the requested Changes
(btw using custom components is like using tailwind with MUI that's Nice combination 😄 )

shivam-bit · 2024-09-08T05:06:14Z

@VipinDevelops LGTM!!🚀🚀

e-for-eshaan · 2024-09-09T05:39:16Z

Big stuff @VipinDevelops 🚀 ! Approved!

samad-yar-khan

LGTM @VipinDevelops

adnanhashmi09

Good stuff vipin 🚀

VipinDevelops marked this pull request as ready for review August 30, 2024 02:18

VipinDevelops changed the title ~~[FEAT]: Service Status~~ [FEAT]: System Status Aug 30, 2024

jayantbh requested review from adnanhashmi09, shivam-bit, e-for-eshaan and samad-yar-khan September 3, 2024 15:10

samad-yar-khan requested changes Sep 6, 2024

View reviewed changes

e-for-eshaan suggested changes Sep 6, 2024

View reviewed changes

e-for-eshaan reviewed Sep 6, 2024

View reviewed changes

VipinDevelops added 12 commits September 7, 2024 01:46

feat: add service page

9361ac2

feat: add reducer for status,logs

b1eadfe

feat: api

c765c6e

feat: add status

8e1cf13

feat: add logs fetch

4c701ee

feat: add webworkers

63738a4

refactor:

6d53a2b

refactor: remove logs

4de238b

refactor: rename to system

cf84f10

refactor:

fb99b26

style:

10beb5f

remove workers

093e1d9

VipinDevelops added 4 commits September 7, 2024 02:09

fix: check fix

6ad0b06

feat:cli color

d66e183

refactor code

7ddeb96

refactor code

fa30a2a

VipinDevelops requested review from samad-yar-khan and e-for-eshaan September 7, 2024 02:58

VipinDevelops changed the title ~~[FEAT]: System Status~~ [FEAT]: System Logs Sep 7, 2024

add eventdata types

9f58ebe

shivam-bit suggested changes Sep 7, 2024

View reviewed changes

VipinDevelops added 2 commits September 7, 2024 13:08

use custom components

c737dce

use span

f7c3ab0

VipinDevelops requested a review from shivam-bit September 7, 2024 07:52

refactor stream backend

50d8550

shivam-bit suggested changes Sep 7, 2024

View reviewed changes

fix style issues

a0d1f24

VipinDevelops requested a review from shivam-bit September 8, 2024 05:02

shivam-bit approved these changes Sep 8, 2024

View reviewed changes

e-for-eshaan approved these changes Sep 9, 2024

View reviewed changes

samad-yar-khan approved these changes Sep 9, 2024

View reviewed changes

adnanhashmi09 approved these changes Sep 10, 2024

View reviewed changes

jayantbh merged commit 592647b into middlewarehq:main Sep 10, 2024
3 checks passed


		const { addPage } = useOverlayPage();

		const ServiceTitle: { [key: string]: string } = {

[FEAT]: System Logs #528

[FEAT]: System Logs #528

Conversation

VipinDevelops commented Aug 30, 2024 • edited Loading

Acceptance Criteria Fulfillment

How it works?

Proposed Changes (including videos or screenshots)

samad-yar-khan commented Sep 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VipinDevelops commented Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e-for-eshaan commented Sep 6, 2024

VipinDevelops commented Sep 6, 2024

e-for-eshaan Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VipinDevelops commented Sep 6, 2024

VipinDevelops commented Sep 6, 2024

e-for-eshaan commented Sep 6, 2024 • edited Loading

VipinDevelops commented Sep 6, 2024

VipinDevelops commented Sep 6, 2024 • edited Loading

VipinDevelops commented Sep 7, 2024 • edited Loading

VipinDevelops commented Sep 7, 2024 • edited Loading

shivam-bit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VipinDevelops Sep 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VipinDevelops commented Sep 7, 2024 • edited by shivam-bit Loading

shivam-bit commented Sep 7, 2024

VipinDevelops commented Sep 7, 2024 • edited Loading

Choose a reason for hiding this comment

VipinDevelops Sep 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VipinDevelops commented Sep 8, 2024 • edited Loading

shivam-bit commented Sep 8, 2024

e-for-eshaan commented Sep 9, 2024

samad-yar-khan left a comment

VipinDevelops commented Aug 30, 2024 •

edited

Loading

VipinDevelops commented Sep 6, 2024 •

edited

Loading

e-for-eshaan Sep 6, 2024 •

edited

Loading

e-for-eshaan commented Sep 6, 2024 •

edited

Loading

VipinDevelops commented Sep 6, 2024 •

edited

Loading

VipinDevelops commented Sep 7, 2024 •

edited

Loading

VipinDevelops commented Sep 7, 2024 •

edited

Loading

VipinDevelops Sep 7, 2024 •

edited

Loading

VipinDevelops commented Sep 7, 2024 •

edited by shivam-bit

Loading

VipinDevelops commented Sep 7, 2024 •

edited

Loading

VipinDevelops Sep 8, 2024 •

edited

Loading

VipinDevelops commented Sep 8, 2024 •

edited

Loading