YARN+Spark an der ETHZ - ETH Zurich | Karteikarten & Zusammenfassungen

Lernmaterialien für YARN+Spark an der ETHZ - ETH Zurich

Greife auf kostenlose Karteikarten, Zusammenfassungen, Übungsaufgaben und Altklausuren für deinen YARN+Spark Kurs an der ETHZ - ETH Zurich zu.

TESTE DEIN WISSEN

With a FAIR scheduler. What happens when everyone tries to access more resources than there are available at the same time?

Lösung anzeigen
TESTE DEIN WISSEN

The one who needs it the most gets it. So the one whos usage is the most different from its instantaneous fair share.

Lösung ausblenden
TESTE DEIN WISSEN


How many instances of a ApplicationMaster are in a cluster in YARN?

Lösung anzeigen
TESTE DEIN WISSEN

Many per cluster, but usually not per every node

Lösung ausblenden
TESTE DEIN WISSEN

What is instantaneous fair share?

Lösung anzeigen
TESTE DEIN WISSEN

The share the remaining users should get if a user (or multiple) are not using the cluster. This should be proportional to their steady fair shares and renormalized to 100%.

Lösung ausblenden
TESTE DEIN WISSEN

List 5 main shortcomings of MapReduce v1, which are addressed by YARN design.

Lösung anzeigen
TESTE DEIN WISSEN
  1. Scalability issues: MapReduce has limited scalability, while YARN can scale to 10,000 nodes.
  2. Rigidity issues: MapReduce v1 only supports MapReduce specific jobs. There is a need, however, for scheduling non-MapReduce workloads. For instance, we would like the ability to share cluster with MPI, graph processing, and any user code.
  3. Resource utilization isues: in MapReduce v1, the reducers wait on the mappers to finish (and vice-versa), leaving large fractions of time when either the reducers or the mappers are idle. Ideally all resources should be used at any given time.
  4. Flexibility issues: mapper and reducer roles are decided at configuration time, and cannot be reconfigured. 
  5. Ability to maintain MapReduce frameworks of different versions.
Lösung ausblenden
TESTE DEIN WISSEN

What are Capacity schedulers?

Lösung anzeigen
TESTE DEIN WISSEN

The CapacityScheduler gives each user certain minimum capacity guarantees. With this strategy cluster resources are allocated over a set of predetermined queues. Each queue gets only a fraction of the cluster resources. As a result, each queue has a minimum guaranteed resource allocation.

Lösung ausblenden
TESTE DEIN WISSEN

Communications between the ResourceManager and NodeManagers are heartbeat-based.

Lösung anzeigen
TESTE DEIN WISSEN

True

Lösung ausblenden
TESTE DEIN WISSEN

Whose responsibility is providing leases to use containers?

Lösung anzeigen
TESTE DEIN WISSEN

ResourceManager

Lösung ausblenden
TESTE DEIN WISSEN

Does YARN plan to allow applications to only request resources in terms of memory usage and number of CPUs?

Lösung anzeigen
TESTE DEIN WISSEN

Yes

Lösung ausblenden
TESTE DEIN WISSEN

Whose responsibility is asking for resources needed for an application?

Lösung anzeigen
TESTE DEIN WISSEN

ResourceManager

Lösung ausblenden
TESTE DEIN WISSEN

 Whose responsibility is fault Tolerance of running applications?

Lösung anzeigen
TESTE DEIN WISSEN

ResourceManager

Lösung ausblenden
TESTE DEIN WISSEN

Can the container allocation/deallocation take place in a dynamic fashion as the application progresses?

Lösung anzeigen
TESTE DEIN WISSEN

Yes

Lösung ausblenden
TESTE DEIN WISSEN

ResourceManager has the ability to request resources back from a running application.

Lösung anzeigen
TESTE DEIN WISSEN

True

Lösung ausblenden
  • 51667 Karteikarten
  • 1087 Studierende
  • 67 Lernmaterialien

Beispielhafte Karteikarten für deinen YARN+Spark Kurs an der ETHZ - ETH Zurich - von Kommilitonen auf StudySmarter erstellt!

Q:

With a FAIR scheduler. What happens when everyone tries to access more resources than there are available at the same time?

A:

The one who needs it the most gets it. So the one whos usage is the most different from its instantaneous fair share.

Q:


How many instances of a ApplicationMaster are in a cluster in YARN?

A:

Many per cluster, but usually not per every node

Q:

What is instantaneous fair share?

A:

The share the remaining users should get if a user (or multiple) are not using the cluster. This should be proportional to their steady fair shares and renormalized to 100%.

Q:

List 5 main shortcomings of MapReduce v1, which are addressed by YARN design.

A:
  1. Scalability issues: MapReduce has limited scalability, while YARN can scale to 10,000 nodes.
  2. Rigidity issues: MapReduce v1 only supports MapReduce specific jobs. There is a need, however, for scheduling non-MapReduce workloads. For instance, we would like the ability to share cluster with MPI, graph processing, and any user code.
  3. Resource utilization isues: in MapReduce v1, the reducers wait on the mappers to finish (and vice-versa), leaving large fractions of time when either the reducers or the mappers are idle. Ideally all resources should be used at any given time.
  4. Flexibility issues: mapper and reducer roles are decided at configuration time, and cannot be reconfigured. 
  5. Ability to maintain MapReduce frameworks of different versions.
Q:

What are Capacity schedulers?

A:

The CapacityScheduler gives each user certain minimum capacity guarantees. With this strategy cluster resources are allocated over a set of predetermined queues. Each queue gets only a fraction of the cluster resources. As a result, each queue has a minimum guaranteed resource allocation.

Mehr Karteikarten anzeigen
Q:

Communications between the ResourceManager and NodeManagers are heartbeat-based.

A:

True

Q:

Whose responsibility is providing leases to use containers?

A:

ResourceManager

Q:

Does YARN plan to allow applications to only request resources in terms of memory usage and number of CPUs?

A:

Yes

Q:

Whose responsibility is asking for resources needed for an application?

A:

ResourceManager

Q:

 Whose responsibility is fault Tolerance of running applications?

A:

ResourceManager

Q:

Can the container allocation/deallocation take place in a dynamic fashion as the application progresses?

A:

Yes

Q:

ResourceManager has the ability to request resources back from a running application.

A:

True

YARN+Spark

Erstelle und finde Lernmaterialien auf StudySmarter.

Greife kostenlos auf tausende geteilte Karteikarten, Zusammenfassungen, Altklausuren und mehr zu.

Jetzt loslegen

Das sind die beliebtesten YARN+Spark Kurse im gesamten StudySmarter Universum

Spanen

Hochschule Hannover

Zum Kurs
Spanen

Hochschule Hannover

Zum Kurs
SPA

Universität Klagenfurt

Zum Kurs
Spax

HFH Hamburger Fern-Hochschule

Zum Kurs

Die all-in-one Lernapp für Studierende

Greife auf Millionen geteilter Lernmaterialien der StudySmarter Community zu
Kostenlos anmelden YARN+Spark
Erstelle Karteikarten und Zusammenfassungen mit den StudySmarter Tools
Kostenlos loslegen YARN+Spark