Difficulty #91
Comments
|
@Mojo-LB we did some tests with algorithm used in Masari, but currently this task was put on hold until it's confirmed as high prio task |
|
@mbg033 An argument can be made that this currently is a high priority task, looking at several issues with current network, and the implications of leaving things as they are. However, I understand this is not your decision, and I hope the management realizes that responding to market demands is good way to approach issues. |
|
I object difficulty algo changes.
Nothing to be hurried. |
|
@Mojo-LB For info, if you use that difficulty algo, you must change Readme to put copyright notice to Massari project at Readme to avoid copyright issue. It’s really annoying when people go around to try to put their copyright at Readme of other projects. |
|
@bobbieltd Please don't make any comments before checking the linked pull request. |
|
@Mojo-LB Sorry, I think you are promoting zawy12’s new difficulty algo. That requires to add copyright notice to Readme and difficulty.cpp for Masari Project. One person (I guess from Masari Project) have raised copyright issue with DeroProject and I feel really angried about that demand. |
|
|
|
2. I am subscribed to two repos, and I saw comment on 100% of repos I’m watching.
3. My pull request doesn’t need a coder to understand it, or to understand that is not related to Masari or zawy12. Zawy12 was an alternative suggestion.
Please have a look at it.
…________________________________
From: Learner <notifications@github.com>
Sent: Saturday, March 10, 2018 10:26:45 AM
To: graft-project/GraftNetwork
Cc: Mojo; Mention
Subject: Re: [graft-project/GraftNetwork] Difficulty (#91)
@Mojo-LB<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmojo-lb&data=02%7C01%7C%7C3a5d06dba19846196b9b08d58660a993%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636562672072978099&sdata=GnEKJsgp%2Fsy0t2nb4iDP7K8cFQHdrnnevYHWJ1LOO%2FA%3D&reserved=0>
1. Check = Check. Let be it.
2. I feel it’s wrong so if I see it is promoted any where. I’ll give a warning. Warning = warning, not spamming, it’s what I thought. It’s same as your way of thinking. If they didn’t raise their copyright demand, I wouldn’t need to be angried with them. I posted only two new issues at two repos where there is post about this new algo. The other two, it’s my replies like this post.
3. Before posting this, I saw your posts relating to new difficulty algo and I didn’t check your pull requests. At this point, I agree that I’m wrong. I’m not hard coders and I don’t understand much blockchain. However, I still can give out my opinion.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgraft-project%2FGraftNetwork%2Fissues%2F91%23issuecomment-372012984&data=02%7C01%7C%7C3a5d06dba19846196b9b08d58660a993%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636562672072978099&sdata=cDnInAGcysF2mvpTj1RFOBU20eC5738rNF1oQLWcYEw%3D&reserved=0>, or mute the thread<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAh-jjeroKgQ40c5x-51dEaM6_ACKI8Rqks5tc45FgaJpZM4SgvJV&data=02%7C01%7C%7C3a5d06dba19846196b9b08d58660a993%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636562672072978099&sdata=5QSl9jJ3qmAVc1iQjZv5dJc%2FBPEbmzdvkIR2RDVHzJ8%3D&reserved=0>.
|
|
I think I’m very wrong in my over-reactions. I need to double check all infos. |
|
@Mojo-LB I think I was confused between two people which are unrelated 😂. Oufff. |
|
There are now 5 Monero clones using my LWMA algorithm. These charts show how much the coins have improved at stopping "hash attacks" and post-attack delays. Miners with 25x the baseline hash rate can jump on without causing hardly any harm. On the LWMA page you can see links to their code. |
|
The suggestion @Mojo-LB has made in his pull request to simply reduce the window from 720 to 60 is a good one. If Monero/Cryptonote/Forknote coins had used this, they would have never been desperate to find a new algorithm, and so no one would have heard of my algorithms. But the LWMA algorithm is a lot better. |
|
@zawy12 Thank you for your endorsement. I realize that my pull request would be better if done as a planned fork rather than change instantly, however I made it just to show how easy it is to make things better. Graft Team announced they are working on it along with the monero7 asic-resistance move, but haven't announced/merged anything yet to indicate if they're simply adjusting the window or using your LWMA algo. https://www.graft.network/2018/03/22/brief-update-status-asic-resistance-difficulty-algorithm/ My personal preference is your LWMA algorithm. It has been proven repeatedly to be the best out there, and I've seen its effect on the stability of Masari network, and several others. |
|
I've been talking to ultranote, and it seems they have 2 hour delays on occasion with N=35. So maybe it is important to go to LWMA |
|
It is. Although adjusting difficulty window makes it much less profitable for anyone to perform a "high hashrate attack", it doesn't allow the network to efficiently recover following such incident. That is a big part where your algo really shines. Devs already have access to this information for some time, and are aware of the conditions that have been affecting mining since day 1 (difficulty and hash pumps) and day 30 (botnet). I think the real issue here is not HOW, but IF and WHEN. |
|
@bobbieltd No one has to use the Masari version of the algorithm pseudocode. My copyright notice is not important as long as it remains open source as the M.I.T license requires. But I like people to know where the algorithm comes from so that they can check for updated versions. Same thing in regards to including a reference to Masari. There are important reasons for people to know where the code originally comes from that is not related to boosting the dev's reputation or whatever. They may want to be able to do searches that can reveal if someone else (not necessarily Masari) made an improvement or found a security hole. |
|
I hear graft lost 340 blocks in the past hour. Similar things were happening to the 25 coins that have switched to my LWMA difficulty algorithm in the last 2 or 3 months. Coins that have switched or are in process of switching: karbo |
|
This shows graft has been a very typical cryptonote coin, suffering from the slow difficulty at the beginning, then seeming to be OK for a while, and then slowly advancing to the point of "blowing up" from the oscillations. They are switching just in time, although I can see the past week (last plot) was not good.
|
|
This is the chart of what happened immediately after I posted the warning above. Graft "blew up" like the others I've been following. This is from being too slow to respond in the presence of big miners having the option to come and go. Maybe ASICs no longer having Monero are the overwhelming cause of this.
At the worst of it (second-to-last magenta peaks) the on-off mining like this got 500 blocks in 8.3 minutes. This was followed by a 6 hour block, 20 hours to get 10 confirmations. They then switched to LWMA and everything looks fine. I included per-LWMA data from above because I needed to change the scales.
|
|
@zawy12 There are significant benefits to having a slower DAA if your coin is not at risk of 51% attacks. For one, the XMR difficulty is remarkably stable, which means that miners get highly consistent payouts, as well as providing them no incentive to switch between multiple coins following profits. Compare that with LWMA, which has up to 50% difficulty fluctuations due to the Poisson distribution of blocks, and their rationale should become clearer. The same also applies for BTC vs BCH, where BTC has a 2-week DAA, and suffers no ill effects due to its majority hashrate position, as opposed to BCH, which had to switch its DAA. If no miner can achieve 51% attack hashrates, they can't perform timewarp attacks or "block-stealing" attacks effectively, and therefore the benefit of having a stable difficulty outweighs the benefit of preventing these attacks. |
|
Timestamp manipulation: let a miner with 2% of the network hashrate send a timestamp 7000 seconds into the future once every 2 hours to see the effect. I've told miners to stay on graft because they'll get 7 days worth of blocks in 3 hours if and when the timestamp manipulator drops by this coin like he has on 4 others the past 2 months. Terminology: time warp attack as it was originally a problem in BTC and described is not possible on coins with rolling averages. Stable difficulty = unstable network. There is no problem from difficulty jumping all around if it that's the result of it accurately tracking network hashrate. It also achieves more consistent solvetimes. See the blow up above which is the result of trying to achieve a stable difficulty. The goal is not stable difficulty, but to match difficulty to the current hashrate. A 24 hour avg like monero is matching difficulty to the hash rate as it was 12 hours in the past (24 hr average). Miners do not want constant difficulty. They want a fair difficulty. Constant difficulty = unfair mining conditions. A difficulty that matches hashrate = a consistently fair difficulty. Talk to miners on the 10 coins using LWMA. They do not about it rising 3x in 30 minutes in response to a 10x hash attack , or about it dropping 80% in 30 minutes after the attackers leave. They love it. They are the small miners who spread the word about a coin and hold the coins, rather than a big miner looking to sell right away. They do not have a problem with it changing all the time +/- 15% as it tries to find the correct value. |
|
I forgot that they have a limit (10xT) inside the difficulty algorithm also. That means it takes a 15% (or maybe 30% miner with the LWMA) to consistently lower it until the 7200 second limit is reached. I could run the test to know how low they can get it but I think someone is going to start doing it before I get a chance to run the test. ( He has to have > 1/10 Network hashpwer to consistently make it go lower until the 7200 limit is reached whereas with the 1/7 he needs more than 13% to do it) |
Indeed you appear to be several hours too late already: this has been happening to graft since last night, with the +2h forged timestamps hitting approximately every 3 hours. The most recent is four blocks in 66220-66228; another couple at 66197 and 66204; another five forged timestamps between 66145-66154; 1 at 66109; another at 65940; two more at 65910-65911. |
|
I still firmly believe the the correct approach to solve the "Hash attacks" and the threat of 51% attacks is to slightly tweak the PoW to limit the means of acquiring massive amount of hash power |
|
@zawy12 You're right about timewarp attacks being nomenclaturally unsound since they can also be used to refer to an older, different attack. Let's stick to using the term "forged timestamps" instead when talking about the issue currently occurring to low hashrate CN coins. You're also right in that miners want a fair difficulty, and when difficulty is divorced from true hashrate, there is a problem. The issue with using fast DAAs is that any sampling of a Poisson process with small N will result in fluctuations, and the only way to correct for this is to increase N. In fact, one can model this deviation. For a given coin taking N blocks into account, Poisson statistics will mean that the actual standard deviation time of these N blocks will be By setting N too small in the absence of external hash attacks, you end up creating deviations from the true hashrate, instead of protecting the network from deviations. |
|
@jagerman Those are either a wrongly configured node hosts, or a probe. We have no way to identify that at this point. |
|
@jagerman In short, the single bad timestamps are not much of a problem due to the 10xT that is in the code. If it was not there, it would be a big problem like I said 2 posts above. The attack has not yet hit. Long story The single +2 hour timestamps are limited to dropping the difficulty in Graft to about (60-10)/60 = 0.83 of the previous difficulty because there is a 10xT limit in the code rather than the FTL=7200 limit. A 20% miner can drop it to about 50% of the correct difficulty in about 50 blocks, but then the 7200 limit starts blocking him, so he has to go away for up to 2 hours before he can repeat the manipulation. But once he erodes difficulty by "only" 25%, 3x more hash power will switch to the coin, and he'll no longer be able to get the blocks he was hoping for, and he won't be able to manipulate it any further because he'll be below the 10% network hash power required to take advantage of the 10xT limits. A > 50% miner can make the difficulty reduce by (N-7200/T)/N max which is D=0 in this case of N=60 and T=120. He can get about 30 blocks at I think 25% of the original difficulty. He also will have to stop periodically to let chain time go back to being close to normal once his blocks start getting near the 7200. You have to go through the simple but confusing recursive math involved here to confirm my numbers and equation. My experiments indicated these approximate results for all DAs. @imperdin Hash attacks by ASICs and NiceHase are not a problem for LWMA when it's implemented with the tighter 500 second limit. >51% hash rate is only a danger to other aspects of a coin. @plavirudar I've written many times about the 1/SQRT(N) effect, although in a Poisson process like this, the variation is a little bit more. I am the source of SMAs with N=17 that Karbo and Sumo loved for curing their cryptonote hash attack problems and spread to many coins. It was only about 9 months ago I finally looked at the results when trying to help with BCH's new DA and realized it was a bad idea. It wasn't near as good as Digishield which is like N = 4x17 = 68. The >24% variation from N=17 was very harmful to sumo and karbo although they did not seem to realize it. It was the source of constant on-off mining. 3x changes in hashrate 20 times a day was almost typical. But they never locked up either. They just knew they were not being sent off to the Cryptonote slow difficulty graveyard like many others....who either died out or copied them. Karbowanc shows the improvement when switching from SMA N=17 to LWMA N=60. BTW the bad spot in the graph below is what I think will happen to Graft pretty soon. Karbowanec had the 7200 and was sorting timestamps which is effectively the same as Graft's current liability.
LWMA N=60 has a speed of response like SMA N=45. So, you are suggesting I might be still pushing for an N that is too low. I have data to argue against this. HUSH uses Zcash's digishield (N like 68). There are many coins with LWMA N=60ish and they all are performing very well and look like the end of the karbo plot above. Most are smaller than hush so you'd expect hush to be more stable. But here's what Hush's results look like, from choosing an SMA equivlaent N=68, more than 50% slower than LWMA N=60. You can see it's not nearly as well. Like karbo, these are very recent blocks.
For comparison, here's Masari's (Masari is 1/7 the price of HUSH with fewer total coins)
|
|
"they accidentally reverted to a previous algorithm" no with full intent. You spoke to them even once? |
|
" They switched POW during this time, but you can't even tell when" I can. IPBCs problems ended the second the PoW was switched. Same for Haven. There has been ZERO problems with those coins who left the monero PoW. |
|
The references to Dero have been removed. With respect, I didn't come here to publicly discuss difficulty algorithms. I came to ask that references to Dero please be removed, and they have been. Dero makes no claim to the LWMA algorithm. |
|
@sebseb7 I chat with IPBC's F.Hearns. I didn't keep my word to review his paper this past week. He said they thought an unsigned integer times a signed one was causing a problem, so they switched back from a tweaked LWMA to the previous LWMA. He didn't go into any details. What problems were they having? I don't see a problem in the difficulties or solvetimes. I sent him a DM to try to find out the details. |
|
@Mojo-LB Graft did the FTL and MTP adjustments like I was wanting, so you can close this issue. |
|
Thank you @zawy12 for all the help you provided and all the time you spent. Thank you to all the contributors who helped :) |
|
"What problems were they having?" the new implementation caused the deamon to not find any blocks because of a possible overun so that part of the fork was cancelled. The plan was to fork to a new PoW and a new LWMA implementation. Only the PoW fork happened and it was successful. No more hashrate fluctuations at all. |
|
OK, I see the difficulty drop at the end was from implementation of the POW. But the hashrate fluctuations were not causing any problem. The blue peaks did not happen very often, and they are not tall. That means they did not have any delays. The perfect average solvetime shows things were functioning fine. I'll work on finding out what happened. Havendev is looking into it because your PM got him worried. (they copied his code) |
|
"But the hashrate fluctuations were not causing any problem." Not at IPBC thats true. IPBC will work with ANY diff algo... |
|
" Havendev is looking into it because your PM got him worried. " The problem at IPBC should worry him. He said "I also spoke to the algo creator Zawy who double reassured there was no issue." You double reassured when "I'll work on finding out what happened." WIP in production how I like that. |
|
Why do you view hash rate fluctuations as a problem if the solvetimes are OK and the big miners were unable to get cheap blocks? |
|
IPBC problem: Since other coins are not having a problem with it, and it worked fine in their testing, and since no pool switched to the new difficulty at the fork, and no block was found, I think it's speculative to believe the fork was done correctly and to assume there was something wrong with the algorithm. |
|
"Why do you view hash rate fluctuations as a problem if the solvetimes are OK and the big miners were unable to get cheap blocks?" never said hashrate fluctuations are a problem. big miners don't need cheap blocks, they are more effective so then can work with way lower margins. |
|
"since no pool switched to the new difficulty at the fork, and no block was found"
|
|
" I think it's speculative to believe the fork was done correctly and to assume there was something wrong with the algorithm." Why is it speculative? You can get the code from github and replay the chain. |
|
" and to assume there was something wrong with the algorithm" I assume you have never seen that before: 2018-Apr-16 11:16:51.701863 ERROR difficulty overhead. |
|
graft right now also fails to accept valid shares: |
|
You implied hash rate fluctuations are bad with
So, then.....why are you saying POW change was needed by IPBC? What are you saying is improved by changing POW if a coin already has LWMA?
Why do you say that when all the miners were saying no pool would update to the newer LWMA? It just sounded like the fork was not done correctly.
That doesn't determine if the fork method or newer LWMA was the problem. Obviously the newer LWMA was fine because testnet was fine. Obviously the fork was bad because ... well it never forked to run the algorithm like testnet did. |
|
"You implied hash rate fluctuations are bad with" I said the PoW change made the hashrate stable. Not stated my opinion wether this is good or bad. "Why do you say that when all the miners were saying no pool would update to the newer LWMA?" maybe you should ask the pools not the miners. They miners can't know the daemon version the pool is running. " Obviously the newer LWMA was fine because testnet was fine." when you use a testnet with higher difficutly the testnet fails also. " Obviously the fork was bad because ... well it never forked to run the algorithm like testnet did." obviously you are assuming all of this. I have seen logfiles from the two biggest pools, have you? |
Again, why did you come here saying a POW change was needed if unstable hashrate is neither good or bad?
How are you going to force testnet to a higher difficulty when it's using the correct algorithm? |
|
"Again, why did you come here saying a POW change was needed if unstable hashrate is neither good or bad?" those people who want to stabilize graft nethash should consider a PoW change. I don't care wether hashrate is stable or not. "How are you going to force testnet to a higher difficulty when it's using the correct algorithm?" you can manupilate the speed of the clock for example... do I really need to explain this? |
|
Your presumption is that LWMA gave an unjustifiably (wrong) high difficulty due to a code error, and you're going to accelerate the testnet clock in order to give a correct high difficulty? What does that achieve? You claim to have data to prove what you're saying. What were the timestamps sent to the LWMA algorithm to make it throw an unusually high difficulty? What was the value of that high difficulty? What difficulty did the pools think was the correct one, and based on what timestamps? We need some sort of data to back up your speculations. |
Big miners need cheap blocks as much as (or more so than) the average miner, since they're doing it for profit instead of doing it for a hobby. |
|
"Your presumption is that LWMA gave an unjustifiably (wrong) high difficulty due to a code error" no I said the IPBC daemon failed at fork time, then LWMA was removed. Thats all I said. Also I don't need to prove it because everyone interested can look for themself. The code is available, the blockchain also. " We need some sort of data to back up your speculations." You said you love Puzzles. Let's have some fun... you don't even have an idea of the size of the puzzle as of now. |
It seems you've reversed course away from promoting a POW change as a cure-all, and now it seems you're reluctant to directly point the finger at the newer LWMA code as the cause of IPBC's unique fork problem. |
|
I don't know what went wrong at IPBC and why problems gone now, you can research for yourself. But fact is, graft may never reach fork height, because it is broken:
|
|
😆 solvetime is infinite now...
|
|
"It seems you've reversed course away from promoting a POW change as a cure-all" I still promote it as a "cure" to the current situation. If you consider the current situation acceptable, then no cure is needed. |
|
I post this to wrong issue. |
Dear @mbg033 ,
Since you are very active today, care to comment on:
#78
For a completely different approach, please check:
zawy12/difficulty-algorithms#3
Thank you :)
The text was updated successfully, but these errors were encountered: