天道酬勤,学无止境

distributed-system

商品 linux 存储场的最佳分布式文件系统 [关闭](Best distributed filesystem for commodity linux storage farm [closed])

问题 关闭。 这个问题是基于意见的。 它目前不接受答案。 想改善这个问题吗? 更新问题,以便通过编辑这篇文章用事实和引文来回答问题。 7年前关闭。 改进这个问题 我有很多闲置的英特尔 linux 服务器(数百个),并希望将它们用于网络托管和文件共享环境中的分布式文件系统。 这不适用于 HPC 应用程序,因此高性能并不重要。 主要要求是高可用性,如果一台服务器离线,存储在其硬盘驱动器上的数据仍然可以从其他节点访问。 它必须通过 TCP/IP 运行并提供标准的 POSIX 文件权限。 我查看了以下内容: Lustre (http://wiki.lustre.org/index.php?title=Main_Page):非常接近,但它没有为节点上的数据提供冗余。 您必须使用 RAID 或 DRBD 使数据 HA。 由 Sun 和 Open Source 支持,所以它应该会存在一段时间 gfarm (http://datafarm.apgrid.org/):看起来它提供了冗余,但以复杂性和可维护性为代价。 支持不如 Lustre。 有没有人对这些或任何其他可能工作的系统有任何经验? 回答1 还检查 GlusterFS 编辑(2012 年 8 月): Ceph 终于准备好了。 最近,作者成立了 Inktank,这是一家独立公司,为其销售商业支持。 根据一些介绍,可挂载的 POSIX

2021-06-12 19:44:34    分类:技术分享    linux   filesystems   distributed-computing   distributed-system

基于paxos的复制键值存储的leader选举(Leader election for paxos-based replicated key value store)

问题 我将使用多 Paxos 实现一个键值存储。 我会有几个节点,其中一个是主节点。 该主节点接收更新请求并将值复制到从节点。 我的问题是如何选择主节点(或领导者)? 我还可以使用 Paxos 算法吗? 如果是这样,您认为是否有必要将 paxos 实现抽象为一个单元,不仅可以被复制单元使用,还可以被领导选举单元使用? 如果我使用 id 最小的节点作为领导者? 如何实施主租约? 感谢您提供任何答案。 回答1 在我进入实际问题之前,我会建议,对于类似 paxos 的系统,您不要将其视为主从关系,而是一种平等的关系。 基本的 Paxos 甚至没有领导者的概念。 Multi-paxos 将领导者作为性能优化,选举领导者是协议的一部分。 Multi-Paxos 归结为下面的 Paxos:有一个准备阶段和一个接受阶段。 Multi-Paxos 的见解是,一旦一个节点赢得了一个接受轮,它同时赢得了领导者选举,之后该领导者不需要准备阶段,直到它检测到另一个节点已经接管了领导权。 现在有一些实用的建议。 我在多个 paxos、multi-paxos 和其他共识系统方面有多年的工作经验。 我首先建议不要实现 Paxos 或 Multi-paxos。 在保持正确性的同时优化 Paxos 系统的性能是非常困难的——尤其是当你遇到这些类型的问题时。 相反,我会考虑实施 Raft 协议。

2021-06-11 17:58:23    分类:技术分享    distributed-system   key-value-store   paxos

查找前 10 个最常访问的 URl,数据跨网络存储(Find Top 10 Most Frequent visited URl, data is stored across network)

问题 来源:谷歌面试题 给定一个大型计算机网络,每台计算机都保存访问过的 url 的日志文件,找到前十个访问量最大的 URL。 有很多大的<string (url) -> int (visits)> maps 。 计算< string (url) -> int (sum of visits among all distributed maps) ,得到组合地图的前十名。 主要限制:地图太大而无法通过网络传输。 也不能直接使用 MapReduce。 我现在遇到了很多此类问题,需要在大型分布式系统上完成处理。 我无法思考或找到合适的答案。 我能想到的只是蛮力,它以某种方式或其他方式违反了给定的约束。 回答1 它说你不能直接使用 map-reduce 这暗示问题的作者希望你思考 map reduce 是如何工作的,所以我们将模仿 map-reduce 的操作: 预处理:让 R 为集群中的服务器数量,从 0,1,2,...,R-1 给每个服务器唯一的 id (map) 对于每个 (string,id) - 将元组发送到具有 id hash(string) % R 。 (减少)完成第 2 步(简单控制通信)后,生成每个服务器前 10 个字符串的(string,count) 。 请注意,那些在步骤 2 中发送到该特定服务器的元组。 (map) 每台服务器将他所有的前 10 名发送给 1

2021-06-10 13:30:03    分类:技术分享    algorithm   performance   distributed-system   large-data

Understanding Gossip protocol

I'm reading akka cluster documentation and now I'm at the Gossip section. I did not understand the following statement: Cluster membership is communicated using a Gossip Protocol, where the current state of the cluster is gossiped randomly through the cluster, with preference to members that have not seen the latest version. It is pretty hard to imagine. I have the following question: Question: how does a node know which members have not seen the latest change if the latest change is still gossiped. I mean, if a node received a notification how it decide where to send it? Obviously, it should

2021-06-10 12:55:36    分类:问答    java   scala   akka   distributed-system   Gossip

PBFT: Why cant the replicas perform the request after 2/3 have prepared? why do we need commit phase?

I know there are some questions on this website that asks the same questions. However the answer is never clear: In PBFT, why cant the replicas execute the requests after 2/3s have prepared? why is commit phase needed? if 2/3 + 1 replica have agreed to prepared, then I owuld think they can execute the request without broadcasting again?

2021-06-03 20:14:06    分类:问答    distributed-system   consensus

Error handling in hadoop map reduce

Based on the documentation, there are a few ways, how the error handling is performed in map reduce. Below are the few: a. Custom counters using enum - increment for every failed record. b. Log error and analyze later. Counters give the number of failed records. However to get the identifier of the failed record(may be its unique key), and details of the exception occurred, node on which the error occurred - we need to perform centralized log analysis and there are many nodes running. Logstash is on which is available. Apart from these, are there any other ways to handle the error scenarios

2021-06-03 11:30:41    分类:问答    hadoop   error-handling   mapreduce   distributed-system

Are PHP sessions hard to scale across a distributed system?

At work we do almost everything in Java and perl, but I wanted to build out a feature using PHP and sessions. Some peeps thought that it was a bad idea to try to do PHP sessions on our system, cause it's distributed to many servers. What would the specific problem be?

2021-06-02 04:28:02    分类:问答    php   session   distributed-system

Leader election for paxos-based replicated key value store

I am going to implement a key value store with multi Paxos. I would have several nodes, one of which is the primary node. This primary node receive update requests and replicate values to slave nodes. My question is how the primary node (or leader) is selected? Can I still use the Paxos algorithm? If so, do you think it is necessary to abstract the paxos implementation to a single unit that could be used not only by the replication unit but also the leader election unit? If I use the node with the least id to be the leader? How can I implement the master lease? Thanks for any answers.

2021-05-31 17:07:58    分类:问答    distributed-system   key-value-store   paxos

What's the difference between ZooKeeper and any distributed Key-Value stores?

I am new to zookeeper and distributed systems, and am learning it myself. From what I understand for now, it seems that ZooKeeper is simply a key-value store whose keys are paths and values are strings, which is nothing different from, say, Redis. (And apparently we can use slash-separated path as keys in redis as well.) So my question is, what is the essential difference between ZooKeeper and other distributed KV store? Why is ZooKeeper using so called "paths" as keys, instead of simple strings?

2021-05-25 22:16:46    分类:问答    distributed   apache-zookeeper   key-value-store   distributed-system

Why is 2-phase commit not suitable for a microservices architecture?

I've read a post saying that: We can not implement traditional transaction system like 2 phase commit in micro-services in a distributed environment. I agree completely with this. But it would be great if someone here can explain the exact reason for this. What would be the issues I'm going to face if I'm implementing 2-phase commit with microservices? Thanks in advance

2021-05-24 20:47:23    分类:问答    microservices   distributed-system   distributed-transactions   2phase-commit