Merge pull request #18 from LennartJKlein/exp-heuristic-value

Experiment on scores at random netlists & gates

experiments/netlist-length/README.md

@@ -0,0 +1,106 @@
+# Correlation between score and length of a netlist
+_Published on: 14-12-2017_
+
+## Summary
+We conducted this experiment to get a better understanding on the returned scores of the solutions. We used a set of predefined conditions (similarly to the [original case](http://heuristieken.nl/wiki/index.php?title=Chips_%26_Circuits)). In this experiment the average score of multiple lengths of netlists were compared.
+This experiment showed that the score is always linearly (independent of the amount of gates, or their location, on the board) within the scope of a netlist length from 2 to 50.
+
+See the ‘discussion’ paragraph for future improvements and extensions on this experiment.
+
+## Background
+Our program is able to solve the [6 predefined netlists](http://heuristieken.nl/resources/CC_netlists2.txt) with a length of 30 up to 70 connections.
+These netlists were designed for 2 predefined boards (consisting of 25 and 50 gates).
+
+![Board 1 - 25 gates](http://heuristieken.nl/wiki/images/7/77/Print1.gif "Board 1 - 25 gates")
+![Board 2 - 50 gates](http://heuristieken.nl/wiki/images/1/1d/Print2.gif "Board 2 - 50 gates")
+
+The program we wrote for this case returns a non-deterministic solution (due to some semi-random pruning). To get a better understanding on the returned scores of the solutions, we conducted this experiment.
+## Hypothesis
+The score is measured by the total length of all the drawn paths on the board. The more paths, the higher the score.
+
+H0 = We expect the score of the solution to grow linear when the length of a netlist (the amount of connections that has to be made on the board) increases.
+## Method
+
+The experiment consists of testing different lengths of netlists on different boards.
+To achieve that, we did the following:
+1. For each amount of gates (30, 35, 40, 45, 50 … to 95), generate a board with the gates placed randomly.
+2. For each generated board, generate 50 random netlists of lengths 2, 3, 4, 5, 6 … to 50
+3. _Run our program_ for every generated board:
+	1. Get the average result of solving a netlist with length 2 a hundred times.
+	2. Get the average result of solving a netlist with length 3 a hundred times.
+	3. Get the average result of solving a netlist with length 4 a hundred times.
+	4. …
+	5. Plot the average *score* for every length of netlist
+
+#### Programmatically
+* Use the runXX.py files to automatically run the __main__.py in the concerned folder.
+* a _runXX.py_ file will loop through the files within a map, and will loop through the maps
+
+#### Conditions
+* A board is set at a width of 18, a height of 17, a depth of 8 (1 + 7 layers).
+* A gate cannot have more than 5 connections.
+* The maximum ‘no improve’ of a solution is set at 5.
+* Only the average score of the 50 randomly generated netlists (of the same length) will be used as results for the experiment
+
+## Results
+Scores at 30 gates
+![Scores 30 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run30_score.png "Scores 30 gates")
+
+Scores at 35 gates
+![Scores 35 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run35_score.png "Scores 35 gates")
+
+Scores at 40 gates
+![Scores 40 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run40_score.png "Scores 40 gates")
+
+Scores at 45 gates
+![Scores 45 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run45_score.png "Scores 45 gates")
+
+Scores at 50 gates
+![Scores 50 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run50_score.png "Scores 50 gates")
+
+Scores at 55 gates
+![Scores 55 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run55_score.png "Scores 55 gates")
+
+Scores at 60 gates
+![Scores 60 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run60_score.png "Scores 60 gates")
+
+Scores at 65 gates
+![Scores 65 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run65_score.png "Scores 65 gates")
+
+Scores at 70 gates
+![Scores 70 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run70_score.png "Scores 70 gates")
+
+Scores at 75 gates
+![Scores 75 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run75_score.png "Scores 75 gates")
+
+Scores at 80 gates
+![Scores 80 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run80_score.png "Scores 80 gates")
+
+Scores at 85 gates
+![Scores 85 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run85_score.png "Scores 85 gates")
+
+Scores at 90 gates
+![Scores 90 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run90_score.png "Scores 90 gates")
+
+Scores at 95 gates
+![Scores 95 gates](https://github.com/LennartJKlein/pathfinder/blob/exp-heuristic-value/experiments/netlist-length/img/run95_score.png "Scores 95 gates")
+
+
+In the graphs above you can see the *average* scores for different lengths of netlists on different amount of gates. Every graph has a linear increase.
+
+## Conclusions
+
+Within the scope of our experiment and by the data we were able to produce, we can say that the score function increases linearly with the length of a netlist. There are two possible explanations for this phenomenon:
+1. The written program always produces the best score for a netlist (which has the shortest possible paths, so the score is only depending on the amount of paths that is drawn)
+2. The score function has a simple nature. If it contained more variables, it could be that the score function behaves differently.
+
+## Discussion
+
+After completing this experiment, we found some ways to improve and extend this experiment in the future:
+1. Increase the scope of the experiment by generating netlists with a length of 50 and up. This can be used to verify or disprove the credibility of this experiment.
+2. Find a way to shorten the runtime of the program (runXX.py) to generate results faster.
+3. Save more data of every solution than purely the score of the board. For example:
+	- if all connections were made
+	- runtime of each solution
+	- amount of layers needed
+	- a heatmap of busy junctions on the board

experiments/netlist-length/generators/auto_gate_generator.py

@@ -0,0 +1,6 @@
+import subprocess
+
+for i in range(1,101):
+	command = "python generate_gates.py " + str(i) + " 100 50"
+	output = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=True)
+	print(output)

experiments/netlist-length/generators/auto_netlist_generator.py

@@ -0,0 +1,6 @@
+import subprocess
+
+for i in range(2,51):
+	command = "python generate_netlists.py " + str(i) + " 100 100" # amount-connections amount-files max-gate-number
+	output = subprocess.check_output(command, stderr=subprocess.STDOUT, shell=True)
+	print(output)

experiments/netlist-length/generators/generate_gates.py

@@ -0,0 +1,76 @@
+"""Sample generators for netlists and gates"""
+import random
+from collections import Counter
+import os
+import numpy as np
+import sys
+
+def make_random_gates(amount_gates, width, height, depth):
+	"""Make a random array of netlists with respect to the contraints.
+
+	:param length: Number of netlists to generate
+	:type length: interger
+
+	:rtype: list of tuples
+    """
+
+	gatesName = []
+	gatesX = []
+	gatesY = []
+	gatesZ = []
+	gates = np.zeros((depth, height, width), dtype=int)
+
+	for i in range(amount_gates):
+
+		random_coord = get_random_coord(width, height, depth)
+		while gates[random_coord[0], random_coord[1], random_coord[2]] != 0:
+			random_coord = get_random_coord(width, height, depth)
+
+		gates[random_coord[0], random_coord[1], random_coord[2]] = i + 1
+		gatesName.append(i)
+		gatesZ.append(random_coord[0])
+		gatesY.append(random_coord[1])
+		gatesX.append(random_coord[2])
+
+	return gatesName, gatesX, gatesY, gatesZ
+
+def get_random_coord(width, height, depth):
+	x = random.sample(range(1, width - 1), 1)
+	y = random.sample(range(1, height - 1), 1)
+	z = random.sample(range(1, depth), 1)
+
+	return (z, y, x)
+
+# Settings
+if len(sys.argv) < 3 or int(sys.argv[1]) <= 0 or int(sys.argv[2]) <= 0:
+	print("ERROR - Correct usage: python generate_gates.py amount-gates amount-files")
+else:
+	amount_gates = int(sys.argv[1])
+	amount_files = int(sys.argv[2])
+	width = 18
+	height = 17
+	depth = 8
+
+	# Create folder
+	os.makedirs("gates/gates-" + str(amount_gates), exist_ok=True)
+
+	# Loop for amount of netlists
+	counter_files = 0
+	while counter_files < amount_files:
+
+		# Create filename
+		filename = "gates/gates-" + str(amount_gates) + "/" + str(counter_files) + ".csv"
+
+		# Write netlist
+		with open(filename, "w") as file:
+			print("name,x,y,z", file=file, end='\n')
+
+			names, xs, ys, zs = make_random_gates(amount_gates + 1, width, height, depth)
+
+			for i in range(1, amount_gates + 1):
+				print(str(names[i]) + "," + str(xs[i])[1:-1] + "," + str(ys[i])[1:-1] + "," + str(zs[i])[1:-1], file=file, end='\n')
+
+		# Count file
+		counter_files += 1
+
+	print("DONE. " + str(counter_files) + " files of "+ str(amount_gates) +" random gates generated.")

experiments/netlist-length/generators/generate_netlists.py

@@ -0,0 +1,77 @@
+"""Sample generators for netlists and gates"""
+import random
+from collections import Counter
+import os
+import sys
+
+def make_random_netlist(length, min_gate, max_gate):
+	"""Make a random array of netlists with respect to the contraints.
+
+	:param length: Number of netlists to generate
+	:type length: interger
+
+	:rtype: list of tuples
+    """
+	netlist = []
+	used_gates = []
+
+	for j in range(length):
+		netlist.append(tuple((random.sample(range(min_gate, max_gate + 1), 2))))
+
+	for tuple_ in netlist:
+		used_gates.append(tuple_[0])
+		used_gates.append(tuple_[1])
+
+	# Make two lists for the length of netlist and number occurrences
+	counter_connections = dict(Counter(netlist))
+	counter_numbers = dict(Counter(used_gates))
+
+	# See if no duplicate connections are made.
+	for gate in counter_connections.values():
+		if gate > 1:
+			return False
+
+	# See if the length of connections is not greater than 5
+	for gate in counter_numbers.values():
+		if gate > 5:
+			return False
+
+	return netlist
+
+
+# Settings
+if len(sys.argv) < 4 or int(sys.argv[1]) <= 0 or int(sys.argv[2]) <= 0:
+	print("ERROR - Correct usage: python generate_netlists.py amount-connections amount-files max-gate-number")
+else:
+	length_netlist = int(sys.argv[1])
+	amount_files = int(sys.argv[2])
+	min_gate = 0
+	max_gate = int(sys.argv[3])
+
+
+	# Create folder
+	os.makedirs("netlists/length100/netlists-" + str(length_netlist), exist_ok=True)
+
+	# Loop for amount of netlists
+	counter_netlists = 0
+	while counter_netlists < amount_files:
+
+		# Create filename
+		filename = "netlists/length100/netlists-" + str(length_netlist) + "/" + str(counter_netlists) + ".txt"
+
+		# Write netlist
+		with open(filename, "w") as file:
+
+			random_netlist = make_random_netlist(length_netlist, min_gate, max_gate)
+			while random_netlist == False:
+				random_netlist = make_random_netlist(length_netlist, min_gate, max_gate)
+
+			print(str(random_netlist), file=file, end='\n')
+
+		# Count netlist
+		counter_netlists += 1
+
+	print("DONE. " + str(counter_netlists) + " netlists of length "+ str(length_netlist) +" generated.")
+
+
+

experiments/netlist-length/img/result run70.txt

@@ -0,0 +1,14 @@
+length: 50 [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100
+.0, 100.0, 100.0, 100.0, 100.0, 100.0, 99.91122448979591, 99.91489795918368, 100.0, 100.0, 99.92448979591838, 99.8542857
+1428572, 100.0, 99.93204081632653, 100.0, 99.8726530612245, 99.87632653061226, 99.93999999999998, 99.88326530612244, 99.
+7730612244898, 99.55857142857144, 99.78530612244897, 100.0, 99.43877551020408, 99.7012244897959, 99.75714285714285, 99.4
+775510204082, 99.4438775510204, 98.82122448979591, 98.4034693877551, 99.08755102040814, 98.98, 99.16734693877552, 98.489
+79591836735]
+score: [31.142857142857142, 44.40816326530612, 58.36734693877551, 75.16326530612245, 86.44897959183673, 101.367346938775
+51, 108.61224489795919, 124.89795918367346, 148.9795918367347, 157.71428571428572, 165.3673469387755, 174.55102040816325
+, 190.16326530612244, 200.79591836734693, 218.3265306122449, 225.9591836734694, 246.6122448979592, 255.6326530612245, 27
+2.7755102040816, 285.83673469387753, 290.1020408163265, 311.1224489795918, 312.6530612244898, 339.734693877551, 359.1224
+489795918, 363.9795918367347, 379.3061224489796, 391.6326530612245, 427.0408163265306, 422.3673469387755, 440.3061224489
+796, 459.7142857142857, 485.9387755102041, 485.40816326530614, 503.48979591836735, 539.2857142857143, 534.7142857142857,
+ 573.2857142857143, 573.1224489795918, 600.1428571428571, 640.7142857142857, 637.9795918367347, 645.1632653061224, 671.8
+979591836735, 694.1836734693877, 702.8775510204082, 718.6938775510204, 743.1836734693877, 753.6734693877551]

experiments/netlist-length/img/result run75.txt

@@ -0,0 +1,13 @@
+length: 50 [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100
+.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 99.91489795918368, 100.0, 99.84285714285714, 100.0, 100.0, 100.0, 100.0, 1
+00.0, 100.0, 99.87632653061223, 100.0, 99.7669387755102, 99.94326530612246, 99.94489795918368, 99.94632653061224, 99.895
+51020408163, 99.64285714285714, 99.70122448979593, 99.70857142857142, 99.85734693877552, 99.67530612244899, 99.365306122
+44898, 99.29081632653059, 99.08755102040814, 99.06469387755102, 99.04244897959185, 98.81632653061224]
+score: [28.183673469387756, 40.816326530612244, 53.673469387755105, 70.93877551020408, 81.6938775510204, 93.122448979591
+84, 106.81632653061224, 125.44897959183673, 127.6734693877551, 142.08163265306123, 156.22448979591837, 161.3877551020408
+, 174.3877551020408, 188.26530612244898, 206.0, 206.10204081632654, 219.59183673469389, 239.14285714285714, 254.63265306
+12245, 268.6530612244898, 274.7755102040816, 283.3265306122449, 297.2040816326531, 307.7142857142857, 321.40816326530614
+, 352.6530612244898, 352.0204081632653, 364.42857142857144, 374.6326530612245, 402.9795918367347, 412.46938775510205, 41
+9.3469387755102, 440.7142857142857, 472.51020408163265, 484.55102040816325, 501.3673469387755, 504.59183673469386, 519.3
+673469387755, 541.5102040816327, 571.9183673469388, 593.9795918367347, 605.8163265306123, 612.8367346938776, 642.7755102
+040817, 659.3265306122449, 676.530612244898, 697.469387755102, 713.5714285714286, 735.0204081632653]

experiments/netlist-length/img/result run80.txt

@@ -0,0 +1,14 @@
+length: 50 [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100
+.0, 100.0, 100.0, 100.0, 100.0, 100.0, 99.82244897959184, 100.0, 100.0, 99.92142857142856, 100.0, 99.92714285714287, 99.
+9295918367347, 99.93204081632653, 100.0, 99.80897959183675, 99.56714285714285, 99.82000000000001, 99.7081632653061, 99.7
+730612244898, 100.0, 99.94632653061224, 99.84326530612245, 99.74489795918367, 99.60163265306123, 99.56285714285713, 99.2
+4020408163267, 99.53653061224492, 99.59224489795919, 99.24673469387756, 99.34836734693877, 98.55489795918368, 98.8759183
+6734696, 98.53061224489795]
+score: [30.857142857142858, 47.42857142857143, 59.08163265306123, 77.34693877551021, 82.42857142857143, 99.6326530612244
+9, 117.59183673469387, 133.18367346938774, 147.08163265306123, 161.53061224489795, 165.26530612244898, 187.8979591836734
+6, 204.75510204081633, 214.0408163265306, 217.6530612244898, 233.77551020408163, 241.6734693877551, 261.6530612244898, 2
+76.6326530612245, 295.3265306122449, 304.16326530612247, 309.3265306122449, 335.7959183673469, 352.1224489795918, 356.53
+061224489795, 379.0816326530612, 390.9183673469388, 393.48979591836735, 426.81632653061223, 429.1020408163265, 456.18367
+346938777, 492.6938775510204, 493.0204081632653, 497.9591836734694, 538.0408163265306, 540.795918367347, 569.59183673469
+39, 571.4897959183673, 598.3265306122449, 631.5510204081633, 638.7551020408164, 654.0612244897959, 671.9795918367347, 71
+4.8163265306123, 720.0816326530612, 734.7755102040817, 730.3469387755102, 758.795918367347, 764.0408163265306]

experiments/netlist-length/img/result run85.txt

@@ -0,0 +1,9 @@
+length: 33 [100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100
+.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0, 99.91489795918368, 100.0, 100.0, 99.92448979591838, 100.0, 99.929591836734
+7, 99.93204081632653, 99.86816326530612, 99.93632653061225, 99.87632653061223]
+score: [29.244897959183675, 47.857142857142854, 58.55102040816327, 73.3265306122449, 87.75510204081633, 98.9387755102040
+8, 115.20408163265306, 126.77551020408163, 141.55102040816325, 157.81632653061226, 163.0204081632653, 176.91836734693877
+, 193.14285714285714, 209.28571428571428, 212.53061224489795, 226.79591836734693, 236.79591836734693, 257.2040816326531,
+ 264.61224489795916, 268.1224489795918, 291.6326530612245, 303.81632653061223, 321.9795918367347, 326.6530612244898, 342
+.48979591836735, 367.2040816326531, 382.0, 384.6530612244898, 405.2448979591837, 416.7551020408163, 439.2244897959184, 4
+50.14285714285717]

experiments/netlist-length/program/gates/gates-1/0.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,3,2,6

experiments/netlist-length/program/gates/gates-1/1.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,8,10,1

experiments/netlist-length/program/gates/gates-1/10.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,9,8,1

experiments/netlist-length/program/gates/gates-1/11.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,15,9,3

experiments/netlist-length/program/gates/gates-1/12.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,5,3,7

experiments/netlist-length/program/gates/gates-1/13.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,4,5,7

experiments/netlist-length/program/gates/gates-1/14.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,11,13,6

experiments/netlist-length/program/gates/gates-1/15.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,4,2,2

experiments/netlist-length/program/gates/gates-1/16.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,16,7,7

experiments/netlist-length/program/gates/gates-1/17.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,16,11,4

experiments/netlist-length/program/gates/gates-1/18.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,5,8,3

experiments/netlist-length/program/gates/gates-1/19.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,6,10,7

experiments/netlist-length/program/gates/gates-1/2.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,14,5,7

experiments/netlist-length/program/gates/gates-1/20.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,14,6,1

experiments/netlist-length/program/gates/gates-1/21.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,10,4,3

experiments/netlist-length/program/gates/gates-1/22.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,14,5,5

experiments/netlist-length/program/gates/gates-1/23.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,13,12,5

experiments/netlist-length/program/gates/gates-1/24.csv

@@ -0,0 +1,2 @@
+name,x,y,z
+1,9,15,5