Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Machine View Sorting / Grouping #9214

Merged
merged 27 commits into from
Jul 14, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
3fa2569
Convert NodeInfo.tsx to a functional component
Jun 21, 2020
51bb292
Update NodeRowGroup to be a functional component
Jun 21, 2020
8ae204e
lint
Jun 21, 2020
265dc75
Convert TotalRow to functional component.
Jun 21, 2020
bd899fe
lint
Jun 22, 2020
9da701d
move node info over to using the sortable table head component. spaci…
Jun 23, 2020
fef027e
Factor a NoewWorkerRow class out of NodeRowGroup that will be usable …
Jun 23, 2020
16e1d51
Compilation checkpoint, I factored the worker filtering logic out of …
Jun 24, 2020
5d10acc
Add sort accessors for CPU
Jun 24, 2020
feb2d18
Add sort accessors for Disk
Jun 24, 2020
8934430
Add sort accessors for RAM
Jun 24, 2020
7493992
add a table sort util for function based accessors (rather than flat …
Jun 24, 2020
2bfa0d3
wip refactor node info features
Jun 24, 2020
bef05fe
wip
Jun 24, 2020
edae34f
Rendering Checkpoint. I've refactored the features and how they are c…
Jun 25, 2020
0d4c423
wip
Jun 26, 2020
69608fa
wip
Jun 26, 2020
c34a2e7
wip
Jun 29, 2020
f7898bb
Finish adding sorting and grouping of machine view
Jun 30, 2020
bdb9502
Merge branch 'master' into machine-view-sort-group
Jun 30, 2020
4b99a42
lint
Jul 6, 2020
a106d86
Merge branch 'master' into machine-view-sort-group
Jul 7, 2020
20c9e09
fix bug in filtration of logs and errors by worker from recent refactor.
Jul 7, 2020
92fb224
Merge branch 'master' into machine-view-sort-group
Jul 9, 2020
f64d92b
Add export of Cluster Disk feature
Jul 10, 2020
b00d42b
Merge branch 'master' into machine-view-sort-group
mfitton Jul 10, 2020
581f490
fix some merge issues
Jul 13, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 2 additions & 10 deletions python/ray/dashboard/client/src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -117,18 +117,10 @@ export type NodeInfoResponse = {
};
load_avg: [[number, number, number], [number, number, number]];
net: [number, number]; // Sent and received network traffic in bytes / second
log_count?: { [pid: string]: number };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we stop collecting this per IP? How do we distinguish logs from same pids in different machines?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. The log entries by IP used to be a top level attribute, so the response looked like

nodeInfoResponse {
clients
log_counts
error_counts
}

where each client in clients represents one IP.

Now, the log_count and error_count exists on a per-client level, so each client has a log_counts and error_counts that corresponds to its IP. This simplifies the logic on the front-end a lot, since trying to refactor the log and error features to work with the NodeFeatures and WorkerFeatures arguments would have been difficult otherwise.

This was changed on the backend accordingly.

error_count?: { [pid: string]: number };
workers: Array<NodeInfoResponseWorker>;
}>;
log_counts: {
[ip: string]: {
[pid: string]: number;
};
};
error_counts: {
[ip: string]: {
[pid: string]: number;
};
};
};

export const getNodeInfo = () => get<NodeInfoResponse>("/api/node_info", {});
Expand Down
72 changes: 44 additions & 28 deletions python/ray/dashboard/client/src/common/SortableTableHead.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -27,49 +27,65 @@ const useSortableTableHeadStyles = makeStyles((theme: Theme) =>
);

export type HeaderInfo<T> = {
id: keyof T;
sortable: boolean;
id: T;
label: string;
numeric: boolean;
};

type SortableTableHeadProps<T> = {
onRequestSort: (event: React.MouseEvent<unknown>, property: keyof T) => void;
onRequestSort: (event: React.MouseEvent<unknown>, id: T) => void;
order: Order;
orderBy: string | null;
orderBy: T | null;
headerInfo: HeaderInfo<T>[];
firstColumnEmpty: boolean;
};

const SortableTableHead = <T,>(props: SortableTableHeadProps<T>) => {
const { order, orderBy, onRequestSort, headerInfo } = props;
const { order, orderBy, onRequestSort, headerInfo, firstColumnEmpty } = props;
const classes = useSortableTableHeadStyles();
const createSortHandler = (property: keyof T) => (
event: React.MouseEvent<unknown>,
) => {
onRequestSort(event, property);
const createSortHandler = (id: T) => (event: React.MouseEvent<unknown>) => {
onRequestSort(event, id);
};
return (
<TableHead>
<TableRow>
{headerInfo.map((headerInfo) => (
<StyledTableCell
key={headerInfo.label}
align={headerInfo.numeric ? "right" : "left"}
sortDirection={orderBy === headerInfo.id ? order : false}
>
<TableSortLabel
active={orderBy === headerInfo.id}
direction={orderBy === headerInfo.id ? order : "asc"}
onClick={createSortHandler(headerInfo.id)}
>
{headerInfo.label}
{orderBy === headerInfo.id ? (
<span className={classes.visuallyHidden}>
{order === "desc" ? "sorted descending" : "sorted ascending"}
</span>
) : null}
</TableSortLabel>
</StyledTableCell>
))}
{firstColumnEmpty && <StyledTableCell />}
{headerInfo.map((headerInfo) => {
if (headerInfo.sortable) {
return (
<StyledTableCell
key={headerInfo.label}
align={headerInfo.numeric ? "right" : "left"}
sortDirection={orderBy === headerInfo.id ? order : false}
>
<TableSortLabel
active={orderBy === headerInfo.id}
direction={orderBy === headerInfo.id ? order : "asc"}
onClick={createSortHandler(headerInfo.id)}
>
{headerInfo.label}
{orderBy === headerInfo.id ? (
<span className={classes.visuallyHidden}>
{order === "desc"
? "sorted descending"
: "sorted ascending"}
</span>
) : null}
</TableSortLabel>
</StyledTableCell>
);
} else {
return (
<StyledTableCell
key={headerInfo.label}
align={headerInfo.numeric ? "right" : "left"}
>
{headerInfo.label}
</StyledTableCell>
);
}
})}
</TableRow>
</TableHead>
);
Expand Down
32 changes: 28 additions & 4 deletions python/ray/dashboard/client/src/common/tableUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,25 @@ export const descendingComparator = <T>(a: T, b: T, orderBy: keyof T) => {
return 0;
};

const descendingComparatorFnAccessor = <T>(
a: T,
b: T,
orderByFn: Accessor<T>,
) => {
const aVal = orderByFn(a);
const bVal = orderByFn(b);
if (bVal < aVal) {
return -1;
}
if (bVal > aVal) {
return 1;
}
return 0;
};

export type Order = "asc" | "desc";
export type Comparator<T> = (a: T, b: T) => number;
export type Accessor<T> = (a: T) => number | string;

export const getComparator = <Key extends keyof any>(
order: Order,
Expand All @@ -22,10 +40,16 @@ export const getComparator = <Key extends keyof any>(
: (a, b) => -descendingComparator(a, b, orderBy);
};

export const stableSort = <T>(
array: T[],
comparator: (a: T, b: T) => number,
) => {
export const getFnComparator = <T>(order: Order, orderByFn: Accessor<T>) => (
a: T,
b: T,
): number => {
return order === "desc"
? descendingComparatorFnAccessor(a, b, orderByFn)
: -descendingComparatorFnAccessor(a, b, orderByFn);
};

export const stableSort = <T>(array: T[], comparator: Comparator<T>) => {
const stabilizedThis = array.map((el, index) => [el, index] as [T, number]);
stabilizedThis.sort((a, b) => {
const order = comparator(a[0], b[0]);
Expand Down
6 changes: 6 additions & 0 deletions python/ray/dashboard/client/src/common/util.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,9 @@ export const getWeightedAverage = (
};

export const sum = (vals: number[]) => vals.reduce((acc, val) => acc + val, 0);

export const filterObj = (obj: Object, filterFn: any) =>
Object.fromEntries(Object.entries(obj).filter(filterFn));

export const mapObj = (obj: Object, filterFn: any) =>
Object.fromEntries(Object.entries(obj).map(filterFn));
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import {
import React, { useState } from "react";
import { connect } from "react-redux";
import { ActorState, RayletActorInfo, RayletInfoResponse } from "../../../api";
import { filterObj } from "../../../common/util";
import { StoreState } from "../../../store";
import Actors from "./Actors";

Expand Down Expand Up @@ -46,9 +47,6 @@ const mapStateToProps = (state: StoreState) => ({
rayletInfo: state.dashboard.rayletInfo,
});

const filterObj = (obj: Object, filterFn: any) =>
Object.fromEntries(Object.entries(obj).filter(filterFn));

type LogicalViewProps = {
rayletInfo: RayletInfoResponse | null;
} & ReturnType<typeof mapStateToProps>;
Expand Down
53 changes: 37 additions & 16 deletions python/ray/dashboard/client/src/pages/dashboard/memory/Memory.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import SortableTableHead, {
} from "../../../common/SortableTableHead";
import { getComparator, Order, stableSort } from "../../../common/tableUtils";
import { StoreState } from "../../../store";
import { dashboardActions } from "../state";
import MemoryRowGroup from "./MemoryRowGroup";
import { MemoryTableRow } from "./MemoryTableRow";

Expand Down Expand Up @@ -50,7 +51,7 @@ const makeGroupedEntries = (
const makeUngroupedEntries = (
memoryTableGroups: MemoryTableGroups,
order: Order,
orderBy: keyof MemoryTableEntry | null,
orderBy: memoryColumnId | null,
) => {
const allEntries = Object.values(memoryTableGroups).reduce(
(allEntries: Array<MemoryTableEntry>, memoryTableGroup) => {
Expand All @@ -71,14 +72,33 @@ const makeUngroupedEntries = (
));
};

const memoryHeaderInfo: HeaderInfo<MemoryTableEntry>[] = [
{ id: "node_ip_address", label: "IP Address", numeric: true },
{ id: "pid", label: "pid", numeric: true },
{ id: "type", label: "Type", numeric: false },
{ id: "object_ref", label: "Object Ref", numeric: false },
{ id: "object_size", label: "Object Size (B)", numeric: true },
{ id: "reference_type", label: "Reference Type", numeric: false },
{ id: "call_site", label: "Call Site", numeric: false },
type memoryColumnId =
| "node_ip_address"
| "pid"
| "type"
| "object_ref"
| "object_size"
| "reference_type"
| "call_site";

const memoryHeaderInfo: HeaderInfo<memoryColumnId>[] = [
{ id: "node_ip_address", label: "IP Address", numeric: true, sortable: true },
{ id: "pid", label: "pid", numeric: true, sortable: true },
{ id: "type", label: "Type", numeric: false, sortable: true },
{ id: "object_ref", label: "Object Ref", numeric: false, sortable: true },
{
id: "object_size",
label: "Object Size (B)",
numeric: true,
sortable: true,
},
{
id: "reference_type",
label: "Reference Type",
numeric: false,
sortable: true,
},
{ id: "call_site", label: "Call Site", numeric: false, sortable: true },
];

const useMemoryInfoStyles = makeStyles((theme: Theme) =>
Expand All @@ -103,9 +123,11 @@ const MemoryInfo: React.FC<{}> = () => {
const { memoryTable, shouldObtainMemoryTable } = useSelector(
memoryInfoSelector,
);
const { setShouldObtainMemoryTable } = useDispatch();
const dispatch = useDispatch();
const toggleMemoryCollection = async () => {
setShouldObtainMemoryTable(!shouldObtainMemoryTable);
dispatch(
dashboardActions.setShouldObtainMemoryTable(!shouldObtainMemoryTable),
);
if (shouldObtainMemoryTable) {
await stopMemoryTableCollection();
}
Expand All @@ -120,9 +142,7 @@ const MemoryInfo: React.FC<{}> = () => {
const [isGrouped, setIsGrouped] = useState(true);
const [order, setOrder] = React.useState<Order>("asc");
const toggleOrder = () => setOrder(order === "asc" ? "desc" : "asc");
const [orderBy, setOrderBy] = React.useState<keyof MemoryTableEntry | null>(
null,
);
const [orderBy, setOrderBy] = React.useState<memoryColumnId | null>(null);
return (
<React.Fragment>
{memoryTable !== null ? (
Expand All @@ -143,9 +163,9 @@ const MemoryInfo: React.FC<{}> = () => {
/>
<Table className={classes.table}>
<SortableTableHead
orderBy={orderBy || ""}
orderBy={orderBy}
order={order}
onRequestSort={(event, property) => {
onRequestSort={(_, property) => {
if (property === orderBy) {
toggleOrder();
} else {
Expand All @@ -154,6 +174,7 @@ const MemoryInfo: React.FC<{}> = () => {
}
}}
headerInfo={memoryHeaderInfo}
firstColumnEmpty={false}
/>
<TableBody>
{isGrouped
Expand Down
Loading